Momentum in large-batch training: Polyak enlarges the critical batch size, Nesterov improves data efficiency

What the math gives to ML

The paper gives a batch-size- and momentum-dependent stability law for one-pass training, rather than treating momentum only as an iteration-acceleration device. Its key transferable asset is the explicit scaling of the largest stable learning rate: Polyak momentum permits a ceiling proportional to B(1-rho), while Nesterov permits a capacity-dependent ceiling proportional to B^beta(1-rho), with beta greater than 1. This suggests a stability-aware large-batch optimizer that scales learning rate and momentum jointly from batch size, while using short online probes to calibrate unknown constants. The approach is most promising for fixed-data-budget pretraining, where the update horizon T=D/B shrinks as batches grow.