Stability Annealing Selects the Implicit Bias of Smoothed Sign Descent: A Rate-Indexed Barrier Path on Separable Data
arXiv:2607.06013
2026
Optimization
1 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper identifies a controllable continuum of implicit biases induced by the stability constant in sign-like optimization: exponentially annealing the denominator regularizer produces a rate-indexed separator rather than either the pure sign or eventual gradient-descent endpoint. Its transferable asset is the exact reinterpretation of smoothed-sign dynamics as entropic mirror ascent, together with a KL-based recursion that gives a quantitative normalized-iterate convergence envelope. The most direct neural-network experiment is an annealed-sign optimizer whose decay rate is treated as an explicit bias knob, evaluated against signSGD and Adam on separable classification and margin-seeking late-training regimes.
Ideas from this paper
✗ Failed on benchmark
2026
Replace the fixed numerical stabilizer in signSGD by an exponentially decaying stability path, so the optimizer remains sign-like for a controllable duration instead of eventually reverting toward ordinary gradient descent as gradients become small. Sweep the decay rate as an explicit implicit-bias parameter: slower annealing should retain the non-Euclidean, barrier-like bias, while faster annealing should approach the sign endpoint more closely.
Useful7/10
Difficulty4/10
Novelty6/10