Vanishing-Perturbation SAM

Integrate this into the SAM optimizer immediately before the perturbed forward/backward pass. For each minibatch, compute the ordinary parameter gradient $g$ at parameters $x$; use a global norm across all trainable parameters, with an epsilon only for numerical safety. Choose $\alpha=1$ for standard SAM or test $\alpha\in\{0.5,1\}$, base radius $\rho$, and clipping coefficient $\tau$. Then execute: (1) $g\leftarrow\nabla f(x)$; (2) $r\leftarrow\min(\rho,\tau(\|g\|+\epsilon)^\alpha)$; (3) perturb every parameter tensor by $\Delta x=r\,g/(\|g\|+\epsilon)^\alpha$; (4) recompute the minibatch gradient $\tilde g=\nabla f(x+\Delta x)$; (5) restore $x$ and apply the usual SAM/AdamW update using $\tilde g$. In pseudocode: g=grad(loss(x)); norm=sqrt(sum(g_i^2)); r=min(rho,tau*(norm+eps)**alpha); for i: p_i += r*g_i/(norm+eps)**alpha; gt=grad(loss(x_perturbed)); restore(); optimizer.step(gt). The mathematical guarantee used by the implementation is $\|\Delta x\|\le\tau\|g\|$, while the threshold $(\rho/\tau)^{1/\alpha}$ is computed exactly from the rule. $L$ need not be known; optionally estimate it with periodic gradient differences $\hat L=\|g_t-g_{t-1}\|/(\|x_t-x_{t-1}\|+\epsilon)$ only for diagnostics. First test on CIFAR-10 with a small ResNet-18 or ViT-Tiny, comparing SGD, AdamW, fixed-radius SAM, and clipped SAM at matched two-forward-pass FLOPs. Track training loss, validation accuracy, gradient norm, and sharpness proxy. The expected signal is that fixed SAM's gradient norm plateaus near a nonzero floor while clipped SAM continues below it without losing SAM's validation advantage; success is lower final loss or accuracy improvement at equal compute, especially when $\rho$ is large enough to make fixed SAM visibly plateau.