△ Mechanism confirmed, baseline not beaten 2026

Exponentially Growing Learning Rate with Update-Norm Restarts

Implementation & benchmark of arXiv:2607.06314 — Gradient descent with exponentially increasing stepsizes and restarts

Usefulness6/10

Difficulty4/10

Novelty5/10

Optimization Speedup Stability Sample efficiency Optimizer Training dynamics Scheduler Optimization Dynamical systems Numerical analysis

Source paper: Gradient descent with exponentially increasing stepsizes and restarts arXiv:2607.06314 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a fixed or hand-tuned learning-rate schedule with a slowly exponentially increasing schedule, and restart the schedule whenever the update norm grows at least as fast as the schedule itself. The restart preserves the current parameters but resets the learning-rate multiplier, allowing the optimizer to repeatedly approach the largest locally stable step size without requiring a Hessian spectrum or a reliable initial learning-rate guess.

Formulas

$$x_{n+1}=x_n-\tau_n\nabla f(x_n),\qquad \tau_n=\tau e^{rn},\quad \tau>0,\;r>0.$$

$$\|x_{n+1}-x_n\|\ge e^r\|x_n-x_{n-1}\|.$$

$$x_{n+1}=\left(1-\tau e^{rn}\right)x_n,\qquad x_{n+1}=\prod_{k=0}^{n}\left(1-\frac{e^{rk}}{10}\right)\quad\text{for }f(x)=x^2/2,\;x_0=1,\;\tau=0.1.$$

$$c(\lambda_{\max},\lambda_{\min},\tau)\ge\frac{3}{2}\log\left|1-\tau\lambda_{\min}\right|,\qquad \lambda_{\min}<\frac{\lambda_{\max}}{2},\quad 0<\tau<\frac{1}{\lambda_{\max}}.$$

Mathematical statement

The paper starts from gradient descent $x_{n+1}=x_n-\tau_n\nabla f(x_n)$ with exponentially increasing stepsizes $\tau_n=\tau e^{rn}$, where $x_n\in\mathbb{R}^d$ is the iterate, $\tau>0$ is the initial stepsize, and $r>0$ is small. It restarts when $\|x_{n+1}-x_n\|\ge e^r\|x_n-x_{n-1}\|$. Since $x_{n+1}-x_n=-\tau_n\nabla f(x_n)$, the criterion compares observed update growth with the known multiplicative growth $e^r$ of the stepsize: growth beyond this threshold signals that the objective dynamics, rather than the schedule alone, are amplifying the update. On the quadratic $f(x)=x^2/2$, the iterates satisfy $x_{n+1}=\left(1-\tau e^{rn}\right)x_n$ and, for $x_0=1$, $x_{n+1}=\prod_{k=0}^{n}(1-e^{rk}/10)$ when $\tau=0.1$. The paper also states that in a quadratic with smallest and largest Hessian eigenvalues $\lambda_{\min}<\lambda_{\max}/2$, the asymptotic rate constant obeys $c(\lambda_{\max},\lambda_{\min},\tau)\ge\frac{3}{2}\log|1-\tau\lambda_{\min}|$ for $0<\tau<1/\lambda_{\max}$. The neural-network adaptation applies the same update-norm inequality to minibatch gradients, with a tolerance factor to avoid false restarts from stochastic noise.

Implementation notes

1. Integration point: implement this as a wrapper around plain minibatch SGD, initially with momentum disabled so the mechanism is isolated. Maintain a scalar schedule clock for the global learning rate. At optimizer step $t$ after a restart, use $\eta_t=\eta_0\exp(rt)$, where $\eta_0$ is deliberately small and $r$ is typically $10^{-3}$ to $10^{-2}$ per optimizer step. Compute the complete parameter update $u_t=-\eta_t g_t$ after gradient clipping but before applying it. 2. Pseudocode: u_prev = None; clock = 0; for batch in loader: g = grad(loss(theta)); eta = eta0*exp(r*clock); u = -eta*g; if u_prev is not None and norm(u) >= beta*exp(r)*norm(u_prev): clock = 0; eta = eta0; u = -eta*g; restart_count += 1; theta += u; u_prev = u; clock += 1. Here beta is a noise guard, initially 1.2 or 1.5. Use a global norm over all parameters, or compare exponentially smoothed norms, s_t = alpha*s_{t-1}+(1-alpha)||u_t||, with alpha around 0.9. In the first experiment, reset momentum or optimizer state at each restart; then run an ablation retaining state. 3. Computed from the paper: the exponential schedule and update-growth inequality. Estimated empirically: update norms, restart frequency, and the locally stable learning-rate scale; no Hessian eigenvalues are needed. 4. First experiment: train a 2-layer MLP and a small ResNet on MNIST or CIFAR-10, comparing cosine decay, fixed tuned SGD, and exponential-restart SGD at equal optimizer steps and FLOPs. Also test synthetic quadratics with condition numbers from $10^2$ to $10^6$. Success means faster loss decrease at equal steps, reduced sensitivity across a $100\times$ range of $\eta_0$, and no sustained divergence. Record the effective peak learning rate before each restart, restart frequency, training loss, validation accuracy, and gradient/update norm trajectories.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built an update-norm restart SGD implementation and verified the scalar exponential-step product formula to numerical precision, with maximum absolute error 5.6e-17. On an ill-conditioned quadratic, exponential growth without restarts diverged with final loss 1.0e191, while the restart rule triggered once and converged to 7.3e-12. The digits MLP did not show a practical training win: fixed SGD reached 89.8% accuracy versus 43.3% for the tested restart configuration.

Agent confidence: 8/10

Baseline: Ill-conditioned quadratic without restarts: final loss 1.0035e191; digits MLP fixed SGD: loss 0.9185, accuracy 89.78%.
Idea: Ill-conditioned quadratic with update-norm restart: final loss 7.334e-12 with 1 restart; digits MLP: loss 2.1459, accuracy 43.33%, 8 restarts, peak learning rate 0.01468; scalar recurrence error 5.55e-17.

Limitations: Only one small digits MLP and one restart hyperparameter configuration were tested; no CIFAR/ResNet, momentum ablation, learning-rate sweep, FLOP-normalized comparison, or broad condition-number study was performed. The neural-network comparison used a tuned fixed learning rate of 0.08 but only one initial rate and tolerance for the restart method.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.06314) Sep 4, 2026

Significantly WORSE than the tuned baseline (+10.0%, p=0.0411, wins 2 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 7.6094
Idea mean: 8.3681
Effect (Δ): 0.7588 (+10.0%; negative = idea better)
Wins: 2 / 8 paired seeds
p-value: 0.0411 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±8.3%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 8 configs

Limitations:

Only the structurally matched tabular track was tested; vision, sequence, and dynamics tracks were not used because this optimizer idea maps to tabular under the bench domain rules. The implementation used plain SGD without momentum, and the idea sweep varied r and lr but held beta at 1.3.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented exponentially growing SGD with update-norm restarts and evaluated it on the structurally matched tabular Friedman#1 regression track using the shared mlp_tiny architecture. Across 8 paired seeds, tuned Adam achieved mean test MSE 7.6094 and restart-SGD achieved 8.3681; the idea was significantly worse with paired delta +0.7588 and permutation p=0.04105. The mechanism signature was observed, but it did not translate into a task-metric win.

Baseline: Adam, best sweep config lr=0.01, weight_decay=0.0001; mean test MSE 7.609379351139069
Idea: Restart SGD, best config lr=0.01, r=0.01, beta=1.3; mean test MSE 8.368132948875427; mean restarts 2.375; observed mean growth-over-schedule 1.0031201995404122

Benchmark result