{
 "artifacts": null,
 "category": "optimization",
 "description": "Replace fixed-beta RMSprop with a bias-corrected, slowly increasing-beta schedule. The paper’s error decomposition contains a stochastic-approximation term of order gamma_n and a memory term of order (1-beta)^2; setting (1-beta_n)^2 proportional to the current learning rate makes the memory contribution decay at the same scale as the optimization error instead of leaving a fixed residual.",
 "formulas_latex": [
  "$$\\theta_n=\\theta_{n-1}+\\gamma_n[\\varepsilon+\\sqrt{v_n}]^{-1}X(\\theta_{n-1},U_n)$$",
  "$$v_n=\\frac{1-\\beta}{1-\\beta^n}\\sum_{k=0}^{n-1}\\beta^kX(\\theta_{n-1-k},U_{n-k})^2$$",
  "$$v_n=\\frac{\\beta(1-\\beta^{n-1})}{1-\\beta^n}v_{n-1}+\\frac{1-\\beta}{1-\\beta^n}X(\\theta_{n-1},U_n)^2$$",
  "$$\\mathbb{E}[f(\\theta_n)]\\lesssim C_0e^{-\\lambda n}+C_1\\gamma_n+C_2(1-\\beta)^2,\\qquad \\rho_n:=1-\\beta_n=\\operatorname{clip}(\\sqrt{c\\gamma_n},\\rho_{\\min},\\rho_{\\max})$$"
 ],
 "id": 2887,
 "implementation": "1. Integration point: replace the second-moment line in RMSprop or Adam’s v-update during training; leave the parameter update and optional decoupled weight decay unchanged. Use one scalar schedule shared by all coordinates. Let g_n be the minibatch gradient and set x_n = -g_n so the sign matches the paper. 2. Pseudocode: initialize v = 0 and bias accumulator q = 1. At step n, obtain g; set gamma = base_lr multiplied by the external learning-rate schedule; set rho = clip(sqrt(c gamma), rho_min, rho_max) and beta = 1 - rho; update v \u003c- beta*v + (1-beta)*g^2; update q \u003c- q*beta; compute the bias-corrected second moment v_hat = v/(1-q); update theta \u003c- theta - gamma*g/(sqrt(v_hat)+epsilon). The bias correction is required because the paper’s normalized finite-history formula assumes a known constant beta, whereas this implementation uses a time-varying beta. 3. What comes from the paper: the exact adaptive denominator recursion, the inverse-moment motivation for safely studying small epsilon, and the decomposition O(gamma_n) + O((1-beta)^2). What is estimated empirically: the constant c, clipping limits, and whether the fixed-beta error decomposition predicts validation behavior; do not estimate Hessians or inverse moments in the first version. 4. Cheap experiment: train a two-layer MLP and a small ResNet-18 on MNIST and CIFAR-10 using SGD, fixed-beta RMSprop with beta in {0.9, 0.99, 0.999}, and the continuation variant with c in {0.01, 0.1, 1}. Compare training loss, validation accuracy, gradient-norm spikes, and sensitivity across epsilon in {0, 10^-12, 10^-8, 10^-4}. Success means lower late-stage loss or validation error at equal optimizer steps, reduced sensitivity to beta and epsilon, and fewer denominator-induced update spikes. Failure is worse early loss without a measurable late-stage benefit.",
 "math_summary": "The RMSprop process uses the coordinatewise stochastic update theta_n = theta_{n-1} + gamma_n [epsilon + sqrt(v_n)]^{-1} X(theta_{n-1}, U_n), where theta_n is the parameter vector, gamma_n \u003e 0 is the step size, epsilon \u003e= 0 is the denominator regularizer, U_n is an independent data/randomness sample, and X(theta,U) is the stochastic update direction; for standard gradient descent, X = -g. For constant beta in [0,1), the normalized second moment is v_n = ((1-beta)/(1-beta^n)) sum_{k=0}^{n-1} beta^k X(theta_{n-1-k},U_{n-k})^2, equivalently v_n = [beta(1-beta^{n-1})/(1-beta^n)]v_{n-1} + [(1-beta)/(1-beta^n)]X(theta_{n-1},U_n)^2. The paper’s main bound is qualitatively of the form E[f(theta_n)] \u003c= C_0 exp(-lambda n) + C_1 gamma_n + C_2(1-beta)^2, with f the objective, lambda \u003e 0 a transient decay rate, and constants C_0,C_1,C_2 uniformly controlled over admissible gamma_n, beta, and epsilon, including epsilon = 0. Adapt this by using time-varying rho_n = 1-beta_n with rho_n^2 = c gamma_n, clipped to [rho_min,rho_max], yielding a memory term with the same target scale as gamma_n. Because the theorem displayed in the extracted material is for fixed beta, the time-varying schedule is an experimentally testable extrapolation rather than a directly proved guarantee.",
 "math_tags": [
  "optimization",
  "probability",
  "stochastic-processes"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.30382",
  "arxiv_url": "https://arxiv.org/abs/2608.30382",
  "summary_what_math_gives_to_ml": "The paper’s transferable asset is not a new preconditioner, but a non-asymptotic analysis showing that RMSprop’s optimization error can be controlled uniformly as the regularization parameter epsilon approaches zero and as the second-moment decay parameter beta approaches its critical value one. In particular, the analysis separates transient error, stochastic-approximation error, and a memory error of order (1-beta)^2, while relying on inverse-moment control of the adaptive denominator. This suggests a principled RMSprop schedule that increases beta during training so the memory error shrinks, while choosing its rate to keep the resulting adaptation lag comparable to the step-size error.",
  "title": "Convergence rates for the RMSprop optimizer with full control of the hyperparameters",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 5,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "accuracy",
  "generalization"
 ],
 "title": "Error-Balanced Beta Continuation",
 "url": "https://synthcore.org/idea/2887/error-balanced-beta-continuation",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}