arXiv:2608.06692 2026 Optimization 2 ideas extracted · analyzed Aug 31, 2026
The paper identifies a concrete failure mode of constant-parameter SAM: the perturbation can prevent convergence below a nonzero gradient norm, with a sharp stationarity scale of $(L\rho)^{1/\alpha}$ for the normalized family. The mechanism is the Lipschitz-gradient error introduced by evaluating the gradient at a displaced point; when this error is comparable to the true gradient, SAM can stabilize at a nonstationary point. The most transferable contribution is a clipped perturbation rule that retains SAM's sharpness exploration at large gradients while making the displacement proportional to the gradient near a solution, restoring convergence to zero. A second practical use is to choose or schedule $\rho$ from a target stationarity tolerance rather than treating it as a scale-free hyperparameter.
Replace constant-radius SAM by a clipped radius that equals the usual radius when the gradient is large but shrinks quickly enough near stationary points. This preserves SAM's sharpness-aware behavior during most training while removing the nonzero stationarity floor caused by a fixed perturbation.
Use the paper's stationarity-floor scale to set the SAM radius from a desired gradient tolerance, and reduce the radius when training approaches that tolerance. This turns an otherwise opaque SAM hyperparameter into a curvature- and accuracy-aware schedule.