# Floor-Aware SAM Radius Scheduling

- ID: 1909
- Canonical URL: https://synthcore.org/idea/1909/floor-aware-sam-radius-scheduling
- API JSON: https://synthcore.org/api/idea/1909.json
- API Markdown: https://synthcore.org/api/idea/1909.md
- Verification status: unverified
- Source: [arXiv:2608.06692](https://arxiv.org/abs/2608.06692)
- Category: optimization
- Solves: stability, accuracy, speedup
- ML areas: optimizer, scheduler, training
- Math tags: optimization, convex-analysis, dynamical-systems
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Use the paper's stationarity-floor scale to set the SAM radius from a desired gradient tolerance, and reduce the radius when training approaches that tolerance. This turns an otherwise opaque SAM hyperparameter into a curvature- and accuracy-aware schedule.

## Mathematical statement

For an $L$-smooth function $f$, the normalized perturbation $y=x+\rho g/\|g\|^\alpha$ has gradient displacement bounded by $L\rho\|g\|^{1-\alpha}$, where $g=\nabla f(x)$ and $L$ is the gradient Lipschitz constant. The ratio of perturbation error to the unperturbed gradient is therefore at most $L\rho/\|g\|^\alpha$. The characteristic floor is $(L\rho)^{1/\alpha}$: below this scale, the perturbation-induced gradient error can be as large as the signal being optimized. Invert this relation for a target tolerance $\varepsilon$: choose $\rho\le\varepsilon^\alpha/L$ if the optimizer must resolve gradients of size $\varepsilon$. Since neural-network curvature varies during training, estimate a conservative local smoothness $\hat L_t$ and use $\rho_t=\min(\rho_{\max},\varepsilon_t^\alpha/(c\hat L_t))$, with safety factor $c>1$. The estimate can use consecutive gradients and parameter changes, $\hat L_t=\|g_t-g_{t-1}\|/(\|x_t-x_{t-1}\|+\epsilon)$, followed by an exponential moving maximum to avoid underestimating curvature.

## Key formulas

- $$\|\nabla f(y)-\nabla f(x)\|\le L\|y-x\|=L\rho\|\nabla f(x)\|^{1-\alpha}.$$
- $$\frac{\|\nabla f(y)-\nabla f(x)\|}{\|\nabla f(x)\|}\le\frac{L\rho}{\|\nabla f(x)\|^\alpha},\qquad \text{floor scale }\|\nabla f(x)\|\approx(L\rho)^{1/\alpha}.$$
- $$\rho_t=\min\left\{\rho_{\max},\frac{\varepsilon_t^\alpha}{c\hat L_t}\right\},\qquad \hat L_t=\frac{\|g_t-g_{t-1}\|}{\|x_t-x_{t-1}\|+\epsilon},\qquad c>1.$$

## Implementation notes

Modify only the radius-selection line of an existing SAM implementation. Maintain the previous parameter vector $x_{t-1}$ and ordinary minibatch gradient $g_{t-1}$, not the perturbed gradient. At step $t$, compute the current ordinary gradient $g_t$ and estimate local smoothness as `Lhat_raw=norm(g_t-g_prev)/(norm(x_t-x_prev)+eps)`. To avoid noisy or artificially small estimates, update `Lhat=beta*Lhat+(1-beta)*Lhat_raw` and clamp it to a running maximum or a conservative lower bound. Choose a target gradient tolerance $\varepsilon_t$: keep it fixed for a simple test, or set it to the running median gradient norm multiplied by a decay factor. Set `rho=min(rho_max, epsilon_t**alpha/(c*Lhat))`, where $c=2$ or $4$ is a safety margin, then perform ordinary SAM with this $\rho_t$: perturb by `rho_t*g_t/(norm(g_t)+eps)**alpha`, recompute the gradient, restore parameters, and update with AdamW or SGD. The paper supplies the inversion $\rho\le\varepsilon^\alpha/L$; the curvature estimate and target schedule are empirical engineering choices. First run a CIFAR-10 ResNet-18 experiment with fixed SAM radius, cosine-decayed radius, and this floor-aware schedule, using the same optimizer and two-pass budget. Evaluate gradient norm at the end of training, validation accuracy, and loss descent per step. The predicted signal is fewer late-training oscillations and a lower final gradient norm than fixed SAM, while preserving its early-training validation improvement. Measure whether reducing perturbation also prevents unstable runs; per-step FLOPs remain approximately unchanged.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.