# Weakly Normally Hyperbolic Cyclic Optimizer

- ID: 1645
- Canonical URL: https://synthcore.org/idea/1645/weakly-normally-hyperbolic-cyclic-optimizer
- API JSON: https://synthcore.org/api/idea/1645.json
- API Markdown: https://synthcore.org/api/idea/1645.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.00812](https://arxiv.org/abs/2608.00812)
- Category: dynamics
- Solves: stability, speedup, generalization
- ML areas: optimizer, scheduler, training-dynamics
- Math tags: dynamical-systems, control-theory, bifurcations, numerical-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Augment an optimizer with a periodic phase and deliberately use a cyclic learning-rate or momentum forcing whose averaged dynamics have an attracting low-dimensional set. Treat the resulting periodic parameter orbit as an invariant torus and tune the schedule so transverse contraction dominates tangential sensitivity and minibatch perturbations. The goal is a robust, phase-locked training orbit that explores parameter space without losing attraction toward a useful solution manifold.

## Mathematical statement

Let $z=(\theta,m)$ denote neural-network parameters and optimizer state, and let $\phi\in\mathbb{T}^1$ be a phase satisfying $\dot\phi=\omega$. The periodically forced training dynamics are $\dot z=f(z,\phi)+\xi(t)$, where $\xi$ represents minibatch noise, and the averaged dynamics are $\dot{\bar z}=\bar f(\bar z)=\frac{1}{2\pi}\int_0^{2\pi}f(\bar z,\phi)\,d\phi$. If the averaged system possesses an attracting invariant manifold $M$ with tangent expansion rate at most $\beta$ and normal contraction rate at least $\alpha>\beta$, then the phase-lifted system has a weakly normally hyperbolic attracting torus. For sufficiently small forcing and perturbation size, a nearby attracting torus persists. The engineering certificate is a positive normal-hyperbolicity gap $\gamma=\alpha-\beta$ and a transverse Floquet multiplier below one.

## Key formulas

- $$\dot z=f(z,\phi)+\xi(t),\qquad \dot\phi=\omega,\qquad \bar f(z)=\frac{1}{2\pi}\int_0^{2\pi}f(z,\phi)\,d\phi.$$
- $$\alpha=-\sup_{(z,\phi)\in\mathcal T}\lambda_{\max}\!\left(\frac{J_N+J_N^{\mathsf T}}{2}\right),\qquad \beta=\sup_{(z,\phi)\in\mathcal T}\lambda_{\max}\!\left(\frac{J_T+J_T^{\mathsf T}}{2}\right),\qquad \alpha>\beta,$$
- $$\|e_N(t)\|\le C e^{-\gamma t}\|e_N(0)\|+\frac{C}{\gamma}\sup_{s\le t}\|\xi_N(s)\|,\qquad \gamma=\alpha-\beta>0,$$
- $$\mu_N=\exp\!\left(\int_0^P\lambda_N(t)\,dt\right)<1,\qquad P=\frac{2\pi}{\omega},$$

## Implementation notes

1. Integration point: modify SGD with momentum or AdamW so the learning rate and momentum are deterministic periodic functions of a phase variable, for example $\eta(\phi)=\eta_0[1+a\sin\phi]$ and $\mu(\phi)=\mu_0+b\cos\phi$, with $\phi_{k+1}=\phi_k+2\pi/P$ modulo $2\pi$. Include optimizer state in $z$, not only weights. First construct the phase-averaged reference optimizer by replacing periodic coefficients with their cycle averages. 2. Pseudocode: initialize $(\theta,m,\phi)$; at each minibatch compute $g=\nabla_\theta L(\theta)$, update $m\leftarrow\mu(\phi)m+g$, update $\theta\leftarrow\theta-\eta(\phi)m$, then update $\phi\leftarrow(\phi+2\pi/P)\bmod 2\pi$. Every $K$ steps, record states at equal phase values and estimate the one-cycle Jacobian using finite differences, automatic Jacobian-vector products, or tangent perturbations. 3. Computed from the mechanism: phase lifting, cycle averaging, tangent/normal decomposition, and the normal-hyperbolicity condition. Estimated empirically: tangent and normal exponents, minibatch perturbation amplitude, and the Floquet multiplier $\mu_N$. Estimate local tangent spaces with PCA on phase-aligned cycle points and measure transverse decay from perturbed replicas. 4. Cheap experiment: train a two-layer MLP on MNIST using constant AdamW, cosine cyclic AdamW, and this phase-augmented optimizer at matched average learning rate and update count. Sweep forcing amplitude $a$ and period $P$. The falsifiable prediction is a stability boundary when the measured transverse multiplier crosses $\mu_N=1$; below the boundary, transverse perturbations decay exponentially with slope approximately $\log|\mu_N|/P$, while above it they fail to decay. The stable region should disappear when the measured gap $\gamma=\alpha-\beta$ becomes comparable to forcing or minibatch-noise amplitude.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a cyclic learning-rate and momentum optimizer toy with phase forcing, periodic-orbit solving, analytic monodromy/Floquet multipliers, finite-difference checks, stability sweeps, noise tests, and a small MLP comparison. Analytic and finite-difference Floquet radii agreed to relative error below 5e-16, and the predicted rho=1 boundary near eta0=2.3224 separated decay from growth with observed slopes close to log(rho). Noise amplification generally increased with rho but was not perfectly monotone, so that prediction is only partially supported.

### Mechanism check

- Verdict: Built a cyclic learning-rate and momentum optimizer toy with phase forcing, periodic-orbit solving, analytic monodromy/Floquet multipliers, finite-difference checks, stability sweeps, noise tests, and a small MLP comparison. Analytic and finite-difference Floquet radii agreed to relative error below 5e-16, and the predicted rho=1 boundary near eta0=2.3224 separated decay from growth with observed slopes close to log(rho). Noise amplification generally increased with rho but was not perfectly monotone, so that prediction is only partially supported.
- Confidence: 8/10
- Limitations: Tests were limited to a 2D linear-quadratic system and a tiny synthetic two-layer regression. MNIST, AdamW, high-dimensional torus/PCA tangent estimation, CUDA, FLOP-matched speed, and repeated statistical generalization studies were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_cyclic_dynamics.py](https://synthcore.org/code/611/bench_cyclic_dynamics.py)
- [bench_report.json](https://synthcore.org/code/611/bench_report.json)
- [cyclic_optimizer_experiment.py](https://synthcore.org/code/611/cyclic_optimizer_experiment.py)
- [report.md](https://synthcore.org/code/611/report.md)
- [report_bench_2026-08-31T195310.md](https://synthcore.org/code/611/report_bench_2026-08-31T195310.md)
- [results.json](https://synthcore.org/code/611/results.json)
- [Download all files as ZIP](https://synthcore.org/download/611)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.