Scaling Limits of Constant-Stepsize SGD at Flat Minima

What the math gives to ML

The paper identifies a concrete failure of the standard constant-step SGD heuristic near flat minima: the stationary error scales as \(\alpha^{1/m}\), not \(\sqrt{\alpha}\), when the Hessian vanishes with local flatness exponent \(m>2\). This gives an actionable rule for selecting stepsizes from a desired stationary-radius budget, together with a mixing-time prediction of order \(\alpha^{-(m-1)}\). The most promising transfer is a flatness-calibrated constant-step optimizer that estimates local curvature decay, chooses \(\alpha\) according to the measured exponent, and optionally uses different stepsizes across parameter blocks. The resulting method is falsifiable through stationary parameter-radius, loss-floor, and equal-FLOP convergence measurements.