Unverified 2026

Floor-Aware SAM Radius Scheduling

Implementation & benchmark of arXiv:2608.06692 — Stationarity Floors and Vanishing Perturbations in Sharpness-Aware Minimization

Usefulness6/10
Difficulty4/10
Novelty6/10

Source paper: Stationarity Floors and Vanishing Perturbations in Sharpness-Aware Minimization arXiv:2608.06692 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the paper's stationarity-floor scale to set the SAM radius from a desired gradient tolerance, and reduce the radius when training approaches that tolerance. This turns an otherwise opaque SAM hyperparameter into a curvature- and accuracy-aware schedule.

Formulas

$$\|\nabla f(y)-\nabla f(x)\|\le L\|y-x\|=L\rho\|\nabla f(x)\|^{1-\alpha}.$$
$$\frac{\|\nabla f(y)-\nabla f(x)\|}{\|\nabla f(x)\|}\le\frac{L\rho}{\|\nabla f(x)\|^\alpha},\qquad \text{floor scale }\|\nabla f(x)\|\approx(L\rho)^{1/\alpha}.$$
$$\rho_t=\min\left\{\rho_{\max},\frac{\varepsilon_t^\alpha}{c\hat L_t}\right\},\qquad \hat L_t=\frac{\|g_t-g_{t-1}\|}{\|x_t-x_{t-1}\|+\epsilon},\qquad c>1.$$

Mathematical statement

For an $L$-smooth function $f$, the normalized perturbation $y=x+\rho g/\|g\|^\alpha$ has gradient displacement bounded by $L\rho\|g\|^{1-\alpha}$, where $g=\nabla f(x)$ and $L$ is the gradient Lipschitz constant. The ratio of perturbation error to the unperturbed gradient is therefore at most $L\rho/\|g\|^\alpha$. The characteristic floor is $(L\rho)^{1/\alpha}$: below this scale, the perturbation-induced gradient error can be as large as the signal being optimized. Invert this relation for a target tolerance $\varepsilon$: choose $\rho\le\varepsilon^\alpha/L$ if the optimizer must resolve gradients of size $\varepsilon$. Since neural-network curvature varies during training, estimate a conservative local smoothness $\hat L_t$ and use $\rho_t=\min(\rho_{\max},\varepsilon_t^\alpha/(c\hat L_t))$, with safety factor $c>1$. The estimate can use consecutive gradients and parameter changes, $\hat L_t=\|g_t-g_{t-1}\|/(\|x_t-x_{t-1}\|+\epsilon)$, followed by an exponential moving maximum to avoid underestimating curvature.

Implementation notes

Modify only the radius-selection line of an existing SAM implementation. Maintain the previous parameter vector $x_{t-1}$ and ordinary minibatch gradient $g_{t-1}$, not the perturbed gradient. At step $t$, compute the current ordinary gradient $g_t$ and estimate local smoothness as Lhat_raw=norm(g_t-g_prev)/(norm(x_t-x_prev)+eps). To avoid noisy or artificially small estimates, update Lhat=beta*Lhat+(1-beta)*Lhat_raw and clamp it to a running maximum or a conservative lower bound. Choose a target gradient tolerance $\varepsilon_t$: keep it fixed for a simple test, or set it to the running median gradient norm multiplied by a decay factor. Set rho=min(rho_max, epsilon_t**alpha/(c*Lhat)), where $c=2$ or $4$ is a safety margin, then perform ordinary SAM with this $\rho_t$: perturb by rho_t*g_t/(norm(g_t)+eps)**alpha, recompute the gradient, restore parameters, and update with AdamW or SGD. The paper supplies the inversion $\rho\le\varepsilon^\alpha/L$; the curvature estimate and target schedule are empirical engineering choices. First run a CIFAR-10 ResNet-18 experiment with fixed SAM radius, cosine-decayed radius, and this floor-aware schedule, using the same optimizer and two-pass budget. Evaluate gradient norm at the end of training, validation accuracy, and loss descent per step. The predicted signal is fewer late-training oscillations and a lower final gradient norm than fixed SAM, while preserving its early-training validation improvement. Measure whether reducing perturbation also prevents unstable runs; per-step FLOPs remain approximately unchanged.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.