# Exponentially Growing Learning Rate with Update-Norm Restarts

- ID: 472
- Canonical URL: https://synthcore.org/idea/472/exponentially-growing-learning-rate-with-update-norm-restart
- API JSON: https://synthcore.org/api/idea/472.json
- API Markdown: https://synthcore.org/api/idea/472.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.06314](https://arxiv.org/abs/2607.06314)
- Category: optimization
- Solves: speedup, stability, sample-efficiency
- ML areas: optimizer, training-dynamics, scheduler
- Math tags: optimization, dynamical-systems, numerical-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10

## Idea description

Replace a fixed or hand-tuned learning-rate schedule with a slowly exponentially increasing schedule, and restart the schedule whenever the update norm grows at least as fast as the schedule itself. The restart preserves the current parameters but resets the learning-rate multiplier, allowing the optimizer to repeatedly approach the largest locally stable step size without requiring a Hessian spectrum or a reliable initial learning-rate guess.

## Mathematical statement

The paper starts from gradient descent $x_{n+1}=x_n-\tau_n\nabla f(x_n)$ with exponentially increasing stepsizes $\tau_n=\tau e^{rn}$, where $x_n\in\mathbb{R}^d$ is the iterate, $\tau>0$ is the initial stepsize, and $r>0$ is small. It restarts when $\|x_{n+1}-x_n\|\ge e^r\|x_n-x_{n-1}\|$. Since $x_{n+1}-x_n=-\tau_n\nabla f(x_n)$, the criterion compares observed update growth with the known multiplicative growth $e^r$ of the stepsize: growth beyond this threshold signals that the objective dynamics, rather than the schedule alone, are amplifying the update. On the quadratic $f(x)=x^2/2$, the iterates satisfy $x_{n+1}=\left(1-\tau e^{rn}\right)x_n$ and, for $x_0=1$, $x_{n+1}=\prod_{k=0}^{n}(1-e^{rk}/10)$ when $\tau=0.1$. The paper also states that in a quadratic with smallest and largest Hessian eigenvalues $\lambda_{\min}<\lambda_{\max}/2$, the asymptotic rate constant obeys $c(\lambda_{\max},\lambda_{\min},\tau)\ge\frac{3}{2}\log|1-\tau\lambda_{\min}|$ for $0<\tau<1/\lambda_{\max}$. The neural-network adaptation applies the same update-norm inequality to minibatch gradients, with a tolerance factor to avoid false restarts from stochastic noise.

## Key formulas

- $$x_{n+1}=x_n-\tau_n\nabla f(x_n),\qquad \tau_n=\tau e^{rn},\quad \tau>0,\;r>0.$$
- $$\|x_{n+1}-x_n\|\ge e^r\|x_n-x_{n-1}\|.$$
- $$x_{n+1}=\left(1-\tau e^{rn}\right)x_n,\qquad x_{n+1}=\prod_{k=0}^{n}\left(1-\frac{e^{rk}}{10}\right)\quad\text{for }f(x)=x^2/2,\;x_0=1,\;\tau=0.1.$$
- $$c(\lambda_{\max},\lambda_{\min},\tau)\ge\frac{3}{2}\log\left|1-\tau\lambda_{\min}\right|,\qquad \lambda_{\min}<\frac{\lambda_{\max}}{2},\quad 0<\tau<\frac{1}{\lambda_{\max}}.$$

## Implementation notes

1. Integration point: implement this as a wrapper around plain minibatch SGD, initially with momentum disabled so the mechanism is isolated. Maintain a scalar schedule clock for the global learning rate. At optimizer step $t$ after a restart, use $\eta_t=\eta_0\exp(rt)$, where $\eta_0$ is deliberately small and $r$ is typically $10^{-3}$ to $10^{-2}$ per optimizer step. Compute the complete parameter update $u_t=-\eta_t g_t$ after gradient clipping but before applying it. 2. Pseudocode: `u_prev = None; clock = 0; for batch in loader: g = grad(loss(theta)); eta = eta0*exp(r*clock); u = -eta*g; if u_prev is not None and norm(u) >= beta*exp(r)*norm(u_prev): clock = 0; eta = eta0; u = -eta*g; restart_count += 1; theta += u; u_prev = u; clock += 1`. Here `beta` is a noise guard, initially 1.2 or 1.5. Use a global norm over all parameters, or compare exponentially smoothed norms, `s_t = alpha*s_{t-1}+(1-alpha)||u_t||`, with `alpha` around 0.9. In the first experiment, reset momentum or optimizer state at each restart; then run an ablation retaining state. 3. Computed from the paper: the exponential schedule and update-growth inequality. Estimated empirically: update norms, restart frequency, and the locally stable learning-rate scale; no Hessian eigenvalues are needed. 4. First experiment: train a 2-layer MLP and a small ResNet on MNIST or CIFAR-10, comparing cosine decay, fixed tuned SGD, and exponential-restart SGD at equal optimizer steps and FLOPs. Also test synthetic quadratics with condition numbers from $10^2$ to $10^6$. Success means faster loss decrease at equal steps, reduced sensitivity across a $100\times$ range of $\eta_0$, and no sustained divergence. Record the effective peak learning rate before each restart, restart frequency, training loss, validation accuracy, and gradient/update norm trajectories.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built an update-norm restart SGD implementation and verified the scalar exponential-step product formula to numerical precision, with maximum absolute error 5.6e-17. On an ill-conditioned quadratic, exponential growth without restarts diverged with final loss 1.0e191, while the restart rule triggered once and converged to 7.3e-12. The digits MLP did not show a practical training win: fixed SGD reached 89.8% accuracy versus 43.3% for the tested restart configuration.

### Mechanism check

- Verdict: Built an update-norm restart SGD implementation and verified the scalar exponential-step product formula to numerical precision, with maximum absolute error 5.6e-17. On an ill-conditioned quadratic, exponential growth without restarts diverged with final loss 1.0e191, while the restart rule triggered once and converged to 7.3e-12. The digits MLP did not show a practical training win: fixed SGD reached 89.8% accuracy versus 43.3% for the tested restart configuration.
- Confidence: 8/10
- Limitations: Only one small digits MLP and one restart hyperparameter configuration were tested; no CIFAR/ResNet, momentum ablation, learning-rate sweep, FLOP-normalized comparison, or broad condition-number study was performed. The neural-network comparison used a tuned fixed learning rate of 0.08 but only one initial rate and tolerance for the restart method.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.04105
- Paired wins: 2/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/1316/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/1316/bench_report.json)
- [experiment.py](https://synthcore.org/code/1316/experiment.py)
- [report.md](https://synthcore.org/code/1316/report.md)
- [report_bench_2026-09-04T142543.md](https://synthcore.org/code/1316/report_bench_2026-09-04T142543.md)
- [results.json](https://synthcore.org/code/1316/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1316)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.