# Continuation-Based Optimizer Stability Map

- ID: 1125
- Canonical URL: https://synthcore.org/idea/1125/continuation-based-optimizer-stability-map
- API JSON: https://synthcore.org/api/idea/1125.json
- API Markdown: https://synthcore.org/api/idea/1125.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.18894](https://arxiv.org/abs/2607.18894)
- Category: dynamics
- Solves: stability, speed
- ML areas: optimizer, training-dynamics, scheduler
- Math tags: dynamical-systems, bifurcations, control-theory, numerical-analysis, linear-algebra
- Ratings: usefulness 8/10; difficulty 7/10; novelty 7/10

## Idea description

Treat the optimizer-plus-network dynamics as a parameterized discrete dynamical system and continue its stationary points as learning rate, momentum, weight decay, or optimizer time constants vary. Detect the transition where a Jacobian eigenvalue crosses the unit circle, then use the computed boundary as an adaptive ceiling instead of discovering instability through failed training.

## Mathematical statement

Let z_k contain network parameters and optimizer states, and let z_(k+1)=F(z_k,p) be one training update with scalar continuation parameter p such as learning rate eta. A stationary training state satisfies G(z,p)=F(z,p)-z=0. Its local stability is determined by J=partial F/partial z: the state is locally asymptotically stable when rho(J)<1, where rho is spectral radius. A discrete Hopf or Neimark-Sacker boundary occurs when a complex-conjugate pair satisfies absolute value(lambda_1,2)=1 while all other eigenvalues remain inside the unit disk. Predictor-corrector continuation tracks solutions of G=0 as p changes. The transferred prediction is that oscillatory training begins at a measurable p_c where rho(J) crosses one, and that perturbations decay geometrically below this boundary.

## Key formulas

- $$z_{k+1}=F(z_k,p),\quad G(z,p)=F(z,p)-z=0.$$
- $$J(z,p)=\frac{\partial F}{\partial z}(z,p),\quad \text{stable iff }\rho(J)<1.$$
- $$|\lambda_{1,2}(J(z,p_c))|=1,\quad \lambda_{1,2}=e^{\pm i\theta},\quad 0<\theta<\pi.$$
- $$\begin{bmatrix}G_z&G_p\\t_z^{\mathsf T}&t_p\end{bmatrix}\begin{bmatrix}\Delta z\\\Delta p\end{bmatrix}=-\begin{bmatrix}G\\0\end{bmatrix}.$$

## Implementation notes

Integration point: use a small MLP and an optimizer with explicit state, such as heavy-ball SGD. Define z=(theta,v), where theta are model parameters and v are momentum variables; begin with a deterministic full-batch loss so that F is reproducible. Choose p=eta, then extend to p=(eta,beta). Pseudocode: initialize a candidate stationary state z near a trained solution; compute G=F(z,p)-z; obtain Jacobian-vector products for G_z and G_p using automatic differentiation; compute a tangent from the nullspace of [G_z,G_p]; take a predictor step along that tangent; Newton-correct the augmented system consisting of G=0 and a pseudo-arclength constraint; estimate the leading eigenvalues of J=partial F/partial z with Arnoldi iteration; stop or reduce p when the leading complex pair reaches modulus one. The paper mechanism supplies equilibrium continuation, spectral stability, and the unit-circle criterion. The implementation estimates the stationary state and Jacobian-vector products. First experiment: use a two-layer MLP on MNIST or a quadratic loss with known Hessian, compare fixed learning-rate sweeps against the continuation map, and measure loss and parameter oscillations. Prediction: persistent oscillations begin near the continued eta_c satisfying rho(J)=1, with measured eta_c within 20 percent of the predicted value; below eta_c, perturbations decay approximately as rho(J)^k.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a deterministic heavy-ball stability-map MVP for a quadratic loss, including the optimizer-state Jacobian, spectral-radius boundary search, decay-rate verification, transition test, and fixed learning-rate sweep. The predicted ceiling eta_c=2(1+beta)/Lambda_max matched numerical unit-circle boundaries for beta=0, 0.5, and 0.9 to machine precision; below the boundary perturbations decayed at the predicted rho within 0.006%, while above it the worst mode diverged strongly. The mechanism is confirmed in this toy setting, but no practical speed improvement is established.

### Mechanism check

- Verdict: Built a deterministic heavy-ball stability-map MVP for a quadratic loss, including the optimizer-state Jacobian, spectral-radius boundary search, decay-rate verification, transition test, and fixed learning-rate sweep. The predicted ceiling eta_c=2(1+beta)/Lambda_max matched numerical unit-circle boundaries for beta=0, 0.5, and 0.9 to machine precision; below the boundary perturbations decayed at the predicted rho within 0.006%, while above it the worst mode diverged strongly. The mechanism is confirmed in this toy setting, but no practical speed improvement is established.
- Confidence: 8/10
- Limitations: This is an exact quadratic toy rather than an MLP/MNIST experiment. It uses direct dense eigendecomposition and an analytic parameter sweep, not full predictor-corrector pseudo-arclength continuation, Arnoldi/JVP estimation, weight decay, stochastic minibatches, or a real wall-clock/FLOP comparison.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.93455
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/217/bench_report.json)
- [continuation_stability.py](https://synthcore.org/code/217/continuation_stability.py)
- [report.md](https://synthcore.org/code/217/report.md)
- [report_bench_2026-08-30T205503.md](https://synthcore.org/code/217/report_bench_2026-08-30T205503.md)
- [results.json](https://synthcore.org/code/217/results.json)
- [stage2_bench.py](https://synthcore.org/code/217/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/217)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.