Stochastic gradient descent with initial regularization

What the math gives to ML

The paper studies a two-phase SGD rule in which isotropic regularization is applied only during an initial training window and then removed, with tail averaging used after the regularization phase. Its transferable asset is not ordinary weight decay itself, but the explicit transient bias-control mechanism: initial shrinkage can suppress unstable high-complexity directions while later unregularized updates recover accuracy, and the analysis gives an operator-level stability condition for the step size. A practical neural-network adaptation is a warm-start regularization schedule with a fixed transition time, followed by tail averaging or stochastic weight averaging. The strongest first test is against constant weight decay, cosine-decayed weight decay, and standard SGD at equal optimizer compute.