Error-Balanced Beta Continuation
Source paper: Convergence rates for the RMSprop optimizer with full control of the hyperparameters arXiv:2608.30382 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace fixed-beta RMSprop with a bias-corrected, slowly increasing-beta schedule. The paper’s error decomposition contains a stochastic-approximation term of order gamma_n and a memory term of order (1-beta)^2; setting (1-beta_n)^2 proportional to the current learning rate makes the memory contribution decay at the same scale as the optimization error instead of leaving a fixed residual.
Formulas
Mathematical statement
The RMSprop process uses the coordinatewise stochastic update theta_n = theta_{n-1} + gamma_n [epsilon + sqrt(v_n)]^{-1} X(theta_{n-1}, U_n), where theta_n is the parameter vector, gamma_n > 0 is the step size, epsilon >= 0 is the denominator regularizer, U_n is an independent data/randomness sample, and X(theta,U) is the stochastic update direction; for standard gradient descent, X = -g. For constant beta in [0,1), the normalized second moment is v_n = ((1-beta)/(1-beta^n)) sum_{k=0}^{n-1} beta^k X(theta_{n-1-k},U_{n-k})^2, equivalently v_n = [beta(1-beta^{n-1})/(1-beta^n)]v_{n-1} + [(1-beta)/(1-beta^n)]X(theta_{n-1},U_n)^2. The paper’s main bound is qualitatively of the form E[f(theta_n)] <= C_0 exp(-lambda n) + C_1 gamma_n + C_2(1-beta)^2, with f the objective, lambda > 0 a transient decay rate, and constants C_0,C_1,C_2 uniformly controlled over admissible gamma_n, beta, and epsilon, including epsilon = 0. Adapt this by using time-varying rho_n = 1-beta_n with rho_n^2 = c gamma_n, clipped to [rho_min,rho_max], yielding a memory term with the same target scale as gamma_n. Because the theorem displayed in the extracted material is for fixed beta, the time-varying schedule is an experimentally testable extrapolation rather than a directly proved guarantee.
Implementation notes
1. Integration point: replace the second-moment line in RMSprop or Adam’s v-update during training; leave the parameter update and optional decoupled weight decay unchanged. Use one scalar schedule shared by all coordinates. Let g_n be the minibatch gradient and set x_n = -g_n so the sign matches the paper. 2. Pseudocode: initialize v = 0 and bias accumulator q = 1. At step n, obtain g; set gamma = base_lr multiplied by the external learning-rate schedule; set rho = clip(sqrt(c gamma), rho_min, rho_max) and beta = 1 - rho; update v <- beta*v + (1-beta)*g^2; update q <- q*beta; compute the bias-corrected second moment v_hat = v/(1-q); update theta <- theta - gamma*g/(sqrt(v_hat)+epsilon). The bias correction is required because the paper’s normalized finite-history formula assumes a known constant beta, whereas this implementation uses a time-varying beta. 3. What comes from the paper: the exact adaptive denominator recursion, the inverse-moment motivation for safely studying small epsilon, and the decomposition O(gamma_n) + O((1-beta)^2). What is estimated empirically: the constant c, clipping limits, and whether the fixed-beta error decomposition predicts validation behavior; do not estimate Hessians or inverse moments in the first version. 4. Cheap experiment: train a two-layer MLP and a small ResNet-18 on MNIST and CIFAR-10 using SGD, fixed-beta RMSprop with beta in {0.9, 0.99, 0.999}, and the continuation variant with c in {0.01, 0.1, 1}. Compare training loss, validation accuracy, gradient-norm spikes, and sensitivity across epsilon in {0, 10^-12, 10^-8, 10^-4}. Success means lower late-stage loss or validation error at equal optimizer steps, reduced sensitivity to beta and epsilon, and fewer denominator-induced update spikes. Failure is worse early loss without a measurable late-stage benefit.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.