△ Mechanism confirmed, baseline not beaten 2026

Continuation-Based Optimizer Stability Map

Implementation & benchmark of arXiv:2607.18894 — Bifurcation Analysis of Sub-Synchronous Oscillations Related to Grid-Forming Converter Inner Controllers

Usefulness8/10

Difficulty7/10

Novelty7/10

Dynamics Stability Speed Optimizer Training dynamics Scheduler Dynamical systems Bifurcations Control theory Numerical analysis Linear algebra

Source paper: Bifurcation Analysis of Sub-Synchronous Oscillations Related to Grid-Forming Converter Inner Controllers arXiv:2607.18894 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Treat the optimizer-plus-network dynamics as a parameterized discrete dynamical system and continue its stationary points as learning rate, momentum, weight decay, or optimizer time constants vary. Detect the transition where a Jacobian eigenvalue crosses the unit circle, then use the computed boundary as an adaptive ceiling instead of discovering instability through failed training.

Formulas

$$z_{k+1}=F(z_k,p),\quad G(z,p)=F(z,p)-z=0.$$

$$J(z,p)=\frac{\partial F}{\partial z}(z,p),\quad \text{stable iff }\rho(J)<1.$$

$$|\lambda_{1,2}(J(z,p_c))|=1,\quad \lambda_{1,2}=e^{\pm i\theta},\quad 0<\theta<\pi.$$

$$\begin{bmatrix}G_z&G_p\\t_z^{\mathsf T}&t_p\end{bmatrix}\begin{bmatrix}\Delta z\\\Delta p\end{bmatrix}=-\begin{bmatrix}G\\0\end{bmatrix}.$$

Mathematical statement

Let z_k contain network parameters and optimizer states, and let z_(k+1)=F(z_k,p) be one training update with scalar continuation parameter p such as learning rate eta. A stationary training state satisfies G(z,p)=F(z,p)-z=0. Its local stability is determined by J=partial F/partial z: the state is locally asymptotically stable when rho(J)<1, where rho is spectral radius. A discrete Hopf or Neimark-Sacker boundary occurs when a complex-conjugate pair satisfies absolute value(lambda_1,2)=1 while all other eigenvalues remain inside the unit disk. Predictor-corrector continuation tracks solutions of G=0 as p changes. The transferred prediction is that oscillatory training begins at a measurable p_c where rho(J) crosses one, and that perturbations decay geometrically below this boundary.

Implementation notes

Integration point: use a small MLP and an optimizer with explicit state, such as heavy-ball SGD. Define z=(theta,v), where theta are model parameters and v are momentum variables; begin with a deterministic full-batch loss so that F is reproducible. Choose p=eta, then extend to p=(eta,beta). Pseudocode: initialize a candidate stationary state z near a trained solution; compute G=F(z,p)-z; obtain Jacobian-vector products for G_z and G_p using automatic differentiation; compute a tangent from the nullspace of [G_z,G_p]; take a predictor step along that tangent; Newton-correct the augmented system consisting of G=0 and a pseudo-arclength constraint; estimate the leading eigenvalues of J=partial F/partial z with Arnoldi iteration; stop or reduce p when the leading complex pair reaches modulus one. The paper mechanism supplies equilibrium continuation, spectral stability, and the unit-circle criterion. The implementation estimates the stationary state and Jacobian-vector products. First experiment: use a two-layer MLP on MNIST or a quadratic loss with known Hessian, compare fixed learning-rate sweeps against the continuation map, and measure loss and parameter oscillations. Prediction: persistent oscillations begin near the continued eta_c satisfying rho(J)=1, with measured eta_c within 20 percent of the predicted value; below eta_c, perturbations decay approximately as rho(J)^k.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a deterministic heavy-ball stability-map MVP for a quadratic loss, including the optimizer-state Jacobian, spectral-radius boundary search, decay-rate verification, transition test, and fixed learning-rate sweep. The predicted ceiling eta_c=2(1+beta)/Lambda_max matched numerical unit-circle boundaries for beta=0, 0.5, and 0.9 to machine precision; below the boundary perturbations decayed at the predicted rho within 0.006%, while above it the worst mode diverged strongly. The mechanism is confirmed in this toy setting, but no practical speed improvement is established.

Agent confidence: 8/10

Baseline: Fixed learning-rate sweep over 20 candidates, including unstable rates; for beta=0.5, rates above eta_c=0.333333 had rho>1 and diverged, with the 1.05*eta_c test reaching maximum loss 7,965,838.81.
Idea: The analytical stability ceiling selected eta=0.8*eta_c=0.266667 for beta=0.5, giving rho=0.707107 and loss after 60 steps of 1.9569e-17. Boundary predictions eta_c=0.222222, 0.333333, and 0.422222 for beta=0, 0.5, and 0.9 matched numerical values with relative error below 2e-16.

Limitations: This is an exact quadratic toy rather than an MLP/MNIST experiment. It uses direct dense eigendecomposition and an analytic parameter sweep, not full predictor-corrector pseudo-arclength continuation, Arnoldi/JVP estimation, weight decay, stochastic minibatches, or a real wall-clock/FLOP comparison.

How to run: python3 continuation_stability.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.18894) Sep 2, 2026

No significant difference vs. the tuned baseline (+1.9%, p=0.9345, wins 5 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0002
Idea mean: 0.0002
Effect (Δ): 0 (+1.9%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.9345 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±66.2%
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

The built-in registered dynamics track was tested with rnn_small, 400 training samples, 200 test samples, 18 epochs, three shared learning-rate configurations, and beta=0.8. Full pseudo-arclength predictor-corrector continuation, Arnoldi eigenvalue tracking, and wall-clock/FLOP comparisons were not tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

The full registered dynamics benchmark found no optimizer win. The stability-capped heavy-ball idea had a confirmed local contraction signature within 9.3% relative error, but tuned Adam was slightly better and the paired permutation test was non-significant.

Baseline: Adam; best_cfg={lr:0.03, weight_decay:0.0}; full 8-seed test MSE mean=0.00024539265723433346, std=0.00009708377932819692.
Comparison: delta_mean=+0.000004588888259604573 (idea minus baseline), p_value=0.93455, idea_wins=5/8, verdict=no significant win.
Idea: Stability-capped heavy-ball; best shared cfg={lr:0.03, weight_decay:0.0}; full 8-seed test MSE mean=0.00024998154549393803, std=0.000093800606773618.

Benchmark result