Scale Weight Decay and Train Better

What the math gives to ML

The paper identifies a principled mismatch in decoupled weight decay: a constant decay coefficient continues shrinking parameters even when the learning rate becomes small, introducing asymptotic bias relative to the unregularized optimizer. Its transferable mechanism is to scale the decay coefficient by the instantaneous learning-rate fraction, so decay is strongest near the peak learning rate and weakens automatically during cooldown and late training. This retains early-training stabilization while reducing persistent parameter-norm drift. The first implementation should modify AdamW or momentum SGD, followed by a matrix-wise optimizer such as Muon.