# Vanishing-Perturbation SAM

- ID: 1908
- Canonical URL: https://synthcore.org/idea/1908/vanishing-perturbation-sam
- API JSON: https://synthcore.org/api/idea/1908.json
- API Markdown: https://synthcore.org/api/idea/1908.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.06692](https://arxiv.org/abs/2608.06692)
- Category: optimization
- Solves: stability, accuracy, generalization
- ML areas: optimizer, training, regularization
- Math tags: optimization, convex-analysis, dynamical-systems
- Ratings: usefulness 8/10; difficulty 4/10; novelty 5/10

## Idea description

Replace constant-radius SAM by a clipped radius that equals the usual radius when the gradient is large but shrinks quickly enough near stationary points. This preserves SAM's sharpness-aware behavior during most training while removing the nonzero stationarity floor caused by a fixed perturbation.

## Mathematical statement

Let $f:\mathbb{R}^n\to\mathbb{R}$ have an $L$-Lipschitz gradient, meaning $\|\nabla f(x)-\nabla f(z)\|\le L\|x-z\|$, and let $g_k=\nabla f(x_k)$. The paper studies $y_k=x_k+\rho g_k/\|g_k\|^\alpha$ for $0\le\alpha\le1$; its displacement norm is $\|y_k-x_k\|=\rho\|g_k\|^{1-\alpha}$. Lipschitz smoothness therefore gives the perturbation-gradient error $\|\nabla f(y_k)-g_k\|\le L\rho\|g_k\|^{1-\alpha}$. Comparing this error with $\|g_k\|$ produces the stationarity scale $(L\rho)^{1/\alpha}$ for $0<\alpha\le1$. We use the clipped radius $\rho_k=\min(\rho,\tau\|g_k\|^\alpha)$, where $\tau>0$ has units of inverse curvature. Then $\|y_k-x_k\|=\rho_k\|g_k\|^{1-\alpha}\le\tau\|g_k\|$, so the perturbation vanishes linearly with the gradient. The resulting update is $x_{k+1}=x_k-h\nabla f(y_k)$, with $h$ chosen below the smoothness stability limit. The constant-radius rule is recovered whenever $\|g_k\|\ge(\rho/\tau)^{1/\alpha}$.

## Key formulas

- $$y_k=x_k+\rho\,\frac{\nabla f(x_k)}{\|\nabla f(x_k)\|^\alpha},\qquad x_{k+1}=x_k-h\nabla f(y_k),\qquad 0\le\alpha\le1.$$
- $$\|y_k-x_k\|=\rho\|\nabla f(x_k)\|^{1-\alpha},\qquad \|\nabla f(y_k)-\nabla f(x_k)\|\le L\rho\|\nabla f(x_k)\|^{1-\alpha}.$$
- $$\|\nabla f(x_k)\|\gtrsim (L\rho)^{1/\alpha}\quad\text{is the attainable stationarity scale for the constant-}\rho\text{ rule}.$$
- $$\rho_k=\min\{\rho,\tau\|g_k\|^\alpha\},\qquad y_k=x_k+\rho_k\frac{g_k}{\|g_k\|^\alpha},\qquad x_{k+1}=x_k-h\nabla f(y_k).$$

## Implementation notes

Integrate this into the SAM optimizer immediately before the perturbed forward/backward pass. For each minibatch, compute the ordinary parameter gradient $g$ at parameters $x$; use a global norm across all trainable parameters, with an epsilon only for numerical safety. Choose $\alpha=1$ for standard SAM or test $\alpha\in\{0.5,1\}$, base radius $\rho$, and clipping coefficient $\tau$. Then execute: (1) $g\leftarrow\nabla f(x)$; (2) $r\leftarrow\min(\rho,\tau(\|g\|+\epsilon)^\alpha)$; (3) perturb every parameter tensor by $\Delta x=r\,g/(\|g\|+\epsilon)^\alpha$; (4) recompute the minibatch gradient $\tilde g=\nabla f(x+\Delta x)$; (5) restore $x$ and apply the usual SAM/AdamW update using $\tilde g$. In pseudocode: `g=grad(loss(x)); norm=sqrt(sum(g_i^2)); r=min(rho,tau*(norm+eps)**alpha); for i: p_i += r*g_i/(norm+eps)**alpha; gt=grad(loss(x_perturbed)); restore(); optimizer.step(gt)`. The mathematical guarantee used by the implementation is $\|\Delta x\|\le\tau\|g\|$, while the threshold $(\rho/\tau)^{1/\alpha}$ is computed exactly from the rule. $L$ need not be known; optionally estimate it with periodic gradient differences $\hat L=\|g_t-g_{t-1}\|/(\|x_t-x_{t-1}\|+\epsilon)$ only for diagnostics. First test on CIFAR-10 with a small ResNet-18 or ViT-Tiny, comparing SGD, AdamW, fixed-radius SAM, and clipped SAM at matched two-forward-pass FLOPs. Track training loss, validation accuracy, gradient norm, and sharpness proxy. The expected signal is that fixed SAM's gradient norm plateaus near a nonzero floor while clipped SAM continues below it without losing SAM's validation advantage; success is lower final loss or accuracy improvement at equal compute, especially when $\rho$ is large enough to make fixed SAM visibly plateau.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built Vanishing-Perturbation SAM with global-gradient clipping and a deterministic toy verification. The mechanism manifested: observed clipping thresholds were within 5.5% of predictions, clipped displacement had the predicted linear gradient slope, the unclipped slopes matched 1-alpha, and the displacement/error bounds held to numerical precision. In the small classifier, clipped SAM reached 0.985 validation accuracy versus 0.990 for fixed SAM and 0.985 for SGD, so the mechanism was verified but no accuracy win over fixed SAM was observed.

### Mechanism check

- Verdict: Built Vanishing-Perturbation SAM with global-gradient clipping and a deterministic toy verification. The mechanism manifested: observed clipping thresholds were within 5.5% of predictions, clipped displacement had the predicted linear gradient slope, the unclipped slopes matched 1-alpha, and the displacement/error bounds held to numerical precision. In the small classifier, clipped SAM reached 0.985 validation accuracy versus 0.990 for fixed SAM and 0.985 for SGD, so the mechanism was verified but no accuracy win over fixed SAM was observed.
- Confidence: 9/10
- Limitations: The mini-experiment used a tiny synthetic 2D MLP rather than CIFAR-10 or ResNet-18, and did not test AdamW, multiple datasets, larger seeds, wall-clock speed, or sharpness proxies. The toy sweep tested alpha=0.5 and 1.0, while the learned-classifier comparison used alpha=1.0 only.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.07195
- Paired wins: 1/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/707/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/707/bench_report.json)
- [experiment.py](https://synthcore.org/code/707/experiment.py)
- [report.md](https://synthcore.org/code/707/report.md)
- [report_bench_2026-09-02T230521.md](https://synthcore.org/code/707/report_bench_2026-09-02T230521.md)
- [results.json](https://synthcore.org/code/707/results.json)
- [Download all files as ZIP](https://synthcore.org/download/707)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.