# Frozen-threshold Adam controller

- ID: 2627
- Canonical URL: https://synthcore.org/idea/2627/frozen-threshold-adam-controller
- API JSON: https://synthcore.org/api/idea/2627.json
- API Markdown: https://synthcore.org/api/idea/2627.md
- Verification status: beats_baseline
- Source: [arXiv:2608.20638](https://arxiv.org/abs/2608.20638)
- Category: optimization
- Solves: stability, accuracy
- ML areas: optimizer, training-dynamics, transformer
- Math tags: dynamical-systems, linear-algebra, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Augment Adam with a layerwise stability monitor based on the paper's normalized frozen stability parameter. Estimate each layer's local sharpness and reduce that layer's learning rate whenever c eta S divided by sqrt(v)+epsilon approaches or exceeds 2. This directly tests whether the one-dimensional edge-of-stability boundary is useful as a safety controller in practical neural-network training.

## Mathematical statement

The paper studies the one-dimensional quadratic f(x)=1/2 x^2 with uncorrected Adam updates. Here x_t is the parameter, m_t is the first-moment state, v_t is the second-moment state, eta is the learning rate, beta_1 and beta_2 are decay factors in [0,1), and epsilon is the positive numerical stabilizer. Define c=(1-beta_1)/(1+beta_1). If the second moment is frozen at v>=0, the update on (x_t,m_t) is linear and its spectral radius is below one exactly when w=c eta/(sqrt(v)+epsilon)<2. For a neural-network direction with local curvature S, use the adapted diagnostic w=c eta S/(sqrt(v)+epsilon). The theorem directly proves the unit-curvature case S=1; layerwise curvature estimates are an empirical extension. Use a safety target 2-delta, with delta>0, rather than operating exactly at the boundary.

## Key formulas

- $$m_{t+1}=\beta_{1}m_{t}+(1-\beta_{1})x_{t},\qquad v_{t+1}=\beta_{2}v_{t}+(1-\beta_{2})x_{t}^{2},\qquad x_{t+1}=x_{t}-\frac{\eta}{\sqrt{v_{t+1}}+\varepsilon}m_{t+1}.$$
- $$c:=\frac{1-\beta_{1}}{1+\beta_{1}},\qquad w_t:=\frac{c\eta}{\sqrt{v_t}+\varepsilon},\qquad w_{\max}:=\frac{c\eta}{\varepsilon}.$$
- $$\mathcal A_v\text{ is spectrally stable}\quad\Longleftrightarrow\quad w:=\frac{c\eta}{\sqrt v+\varepsilon}<2.$$
- $$w_{\ell,t}:=\frac{c\,\eta_{\ell,t}\,\widehat S_{\ell,t}}{\sqrt{\overline v_{\ell,t}}+\varepsilon},\qquad \eta_{\ell,t+1}=\eta_{\ell,t}\min\left(1,\frac{2-\delta}{\max(w_{\ell,t},w_{\min})}\right).$$

## Implementation notes

Integrate this into AdamW at the parameter-group learning-rate line, using one scalar controller per layer or per optimizer parameter group rather than one controller per individual weight. First perform the usual uncorrected Adam moment updates for each parameter tensor theta_l: compute gradient g_l, set m_l to beta_1 m_l plus (1-beta_1) g_l, and set v_l to beta_2 v_l plus (1-beta_2) g_l squared. Maintain a scalar second-moment summary vbar_l, using the mean or median of v_l and optionally smoothing it with an exponential moving average. Every K optimizer steps, estimate local sharpness S_hat_l with one or more Hessian-vector products. Sample a normalized random vector u_l, compute h_l=gradient_theta_l of the scalar product between the full gradient and u_l, and set S_hat_l=absolute value of the scalar product u_l transpose h_l. Smooth this estimate to reduce probe noise. Compute c=(1-beta_1)/(1+beta_1) and w_l=c eta_l S_hat_l/(sqrt(vbar_l)+epsilon). Apply eta_l <- eta_l times min(1,(2-delta)/max(w_l,w_min)), with delta between 0.1 and 0.3. Optionally allow slow recovery, such as eta_l <- min(eta_initial,1.001 eta_l), only when w_l<1 for several consecutive checks. Use the controlled rate in the Adam update and keep decoupled weight decay separate. The theorem supplies the threshold 2; curvature estimation, tensor aggregation, and recovery are empirical extensions. Begin with a six-layer MLP or small ViT on CIFAR-10. Compare AdamW, AdamW with global gradient clipping, and the controller at matched nominal learning rates and FLOPs. Log w_l, loss spikes, gradient norms, and validation accuracy. Success means fewer divergent runs, fewer sharp loss oscillations, and equal-or-better validation accuracy at the same training budget. A stronger result is faster loss descent because the controller permits a larger nominal learning rate without sustained supercritical behavior.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built a frozen-threshold Adam controller with exact frozen-dynamics verification and a matched small MLP experiment. The mechanism manifested: measured stability boundaries were within 0.05% of the predicted w=2 boundary, all 12 curvature/second-moment scaling tests had measured/predicted ratios of 1.0, and supercritical controller updates mapped w=2, 3, and 8 exactly to the 1.8 safety target. In the MLP, w stayed subcritical with maximum 0.0824, so the controller never reduced the learning rate and produced identical results to baseline; this confirms the toy mechanism but not a practical training win.

### Mechanism check

- Verdict: Built a frozen-threshold Adam controller with exact frozen-dynamics verification and a matched small MLP experiment. The mechanism manifested: measured stability boundaries were within 0.05% of the predicted w=2 boundary, all 12 curvature/second-moment scaling tests had measured/predicted ratios of 1.0, and supercritical controller updates mapped w=2, 3, and 8 exactly to the 1.8 safety target. In the MLP, w stayed subcritical with maximum 0.0824, so the controller never reduced the learning rate and produced identical results to baseline; this confirms the toy mechanism but not a practical training win.
- Confidence: 9/10
- Limitations: Only one fixed-seed, 300-step synthetic MLP run was tested; no CIFAR-10, multiple seeds, AdamW weight decay, global clipping comparison, FLOP-matched larger-learning-rate sweep, or divergence-rate study. The neural sharpness estimate used one random Hessian-vector probe per check, and the controller remained inactive because the observed diagnostic was far below threshold.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/1009/bench_report.json)
- [bench_runner.py](https://synthcore.org/code/1009/bench_runner.py)
- [experiment.py](https://synthcore.org/code/1009/experiment.py)
- [report.md](https://synthcore.org/code/1009/report.md)
- [report_bench_2026-09-01T181147.md](https://synthcore.org/code/1009/report_bench_2026-09-01T181147.md)
- [results.json](https://synthcore.org/code/1009/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1009)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.