Convergence rates for the RMSprop optimizer with full control of the hyperparameters
arXiv:2608.30382
2026
Optimization
1 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper’s transferable asset is not a new preconditioner, but a non-asymptotic analysis showing that RMSprop’s optimization error can be controlled uniformly as the regularization parameter epsilon approaches zero and as the second-moment decay parameter beta approaches its critical value one. In particular, the analysis separates transient error, stochastic-approximation error, and a memory error of order (1-beta)^2, while relying on inverse-moment control of the adaptive denominator. This suggests a principled RMSprop schedule that increases beta during training so the memory error shrinks, while choosing its rate to keep the resulting adaptation lag comparable to the step-size error.
Ideas from this paper
Unverified
2026
Replace fixed-beta RMSprop with a bias-corrected, slowly increasing-beta schedule. The paper’s error decomposition contains a stochastic-approximation term of order gamma_n and a memory term of order (1-beta)^2; setting (1-beta_n)^2 proportional to the current learning rate makes the memory contribution decay at the same scale as the optimization error instead of leaving a fixed residual.
Useful6/10
Difficulty4/10
Novelty5/10