# Predictive Lyapunov Switched Optimizer

- ID: 1579
- Canonical URL: https://synthcore.org/idea/1579/predictive-lyapunov-switched-optimizer
- API JSON: https://synthcore.org/api/idea/1579.json
- API Markdown: https://synthcore.org/api/idea/1579.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.29143](https://arxiv.org/abs/2607.29143)
- Category: dynamics
- Solves: stability, speed, accuracy
- ML areas: optimizer, training-dynamics, scheduler
- Math tags: control-theory, dynamical-systems, optimization, linear-algebra
- Ratings: usefulness 8/10; difficulty 6/10; novelty 7/10

## Idea description

Replace a single optimizer update rule by a finite set of update modes, such as conservative SGD, momentum SGD, high-step SGD, and Adam-like preconditioned descent. Because the selected mode is applied with a one-step delay, score every candidate using a nominal predictor and choose the mode with the greatest certified decrease of a phase-dependent Lyapunov function around a short periodic optimizer orbit. This creates a controlled limit cycle in parameter or loss-state space, allowing stable oscillatory training instead of forcing the dynamics toward a stationary point.

## Mathematical statement

Let $z_k$ be a low-dimensional training state containing parameters projected onto a monitored subspace, loss, gradient norm, and optimizer memory; let $j_k\in\{1,\ldots,K\}$ denote the currently active optimizer mode. Approximate each mode locally by the uncertain affine model $z_{k+1}=(\bar A_j+\Delta A_j)z_k+\bar b_j+\Delta b_j$, where $\bar A_j,\bar b_j$ are fitted nominal dynamics and $(\Delta A_j,\Delta b_j)$ belong to an empirically estimated polytope. For cycle phases $i\in\{1,\ldots,N\}$, use centers $\rho_i$ and positive-definite matrices $P_i$ and define $V_i(z)=(z-\rho_i)^\top P_i(z-\rho_i)$. The predictive controller evaluates each candidate mode with the delayed-mode predictor $\hat z_{k+1}^{(j)}=\bar A_jz_k+\bar b_j$ and chooses the mode minimizing the next-phase Lyapunov value. A robust decrease certificate requires $V_{i^+}(z_{k+1})-V_i(z)\leq-\gamma\|z-\rho_i\|_2^2$ for every uncertainty vertex, where $i^+$ is the next cycle phase and $\gamma>0$. If no mode satisfies the condition, choose the candidate with the smallest worst-case violation and reduce its step size.

## Key formulas

- $$V_i(z)=(z-\rho_i)^\top P_i(z-\rho_i),\qquad P_i\succ0,$$
- $$\hat z_{k+1}^{(j)}=\bar A_j z_k+\bar b_j,\qquad j_k=\arg\min_j V_{i^+}\!\left(\hat z_{k+1}^{(j)}\right),$$
- $$V_{i^+}\!\left((\bar A_j+\Delta A_j)z+(\bar b_j+\Delta b_j)\right)-V_i(z)\leq-\gamma\|z-\rho_i\|_2^2\quad\forall(\Delta A_j,\Delta b_j)\in\mathcal V_j,$$
- $$z_{k+1}=\operatorname{Update}_{j_k}(\theta_k,g_k,m_k),\qquad \theta_{k+1}=\theta_k-\eta_{j_k}D_{j_k}(g_k,m_k).$$

## Implementation notes

Integration point: insert the controller immediately before the optimizer step in an ordinary PyTorch training loop. Define three to five modes with identical model parameters but different update rules or learning rates: conservative SGD with learning rate $\eta$, momentum SGD with $2\eta$, Adam with $\eta$, and a damping mode with $\eta/4$. Maintain a compact state $z_k$ rather than a full parameter-space model: concatenate current loss, log gradient norm, cosine similarity between successive gradients, normalized update norm, and a random projection of the parameter vector. First collect trajectories under each mode separately and fit $\bar A_j,\bar b_j$ by ridge regression from $(z_k,z_{k+1})$. Estimate uncertainty by fitting residual ellipsoids or coordinate-wise boxes on held-out trajectories; use their vertices as $\mathcal V_j$. Choose a two- or three-phase cycle by clustering recurring states, or initialize a two-phase cycle from low-loss and recovery states. Solve for diagonal or full $P_i\succ0$ using a small semidefinite program over sampled uncertainty vertices, enforcing the displayed decrease inequality. If an SDP is inconvenient, optimize Cholesky factors of $P_i$ with hinge penalties. Because the controller is delayed, queue the selected mode and apply the previously queued mode at the next step. At each iteration, observe $z_k$, predict $\hat z_{k+1}^{(j)}$ for every mode, compute all vertex Lyapunov differences, select the mode with the smallest worst-case score, queue it, apply the previous queued mode, update the network, and log $V_i$ and certificate violations. The paper supplies the phase-dependent Lyapunov and robust-vertex logic; the neural-network state, affine models, uncertainty sets, cycle centers, and optimizer modes are estimated empirically. The first cheap experiment should use a two-layer MLP on MNIST, comparing Adam, SGD, cosine-scheduled SGD, and the switched controller at matched average update cost. Deliberately test learning rates around the fixed-mode instability boundary. The quantitative prediction is that a fixed mode becomes unstable when its local predicted closed-loop spectral radius crosses one, approximately $\rho(\bar A_j)\approx1$. With switching active, phase-conditioned Lyapunov drift should be negative outside a residual uncertainty set, and $\|z_k-\rho_{i_k}\|$ should decay geometrically until reaching the uncertainty floor. Test whether the empirical divergence threshold is predicted within 20% by the fitted spectral radius and whether switched training remains bounded beyond the best fixed-mode threshold.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a predictive Lyapunov switched affine-dynamics prototype with delayed mode application, affine identification, uncertainty vertices, and deterministic parameter sweeps. The fixed-mode spectral boundary and oracle alternating-cycle decay matched predictions reasonably: fitted spectral radius was 1.04 versus true 1.04, and the observed oracle two-step ratio was 0.78 versus predicted 0.78. However, the actual delayed Lyapunov argmin did not select the stabilizing cycle, so the proposed controller mechanism was not validated.

### Mechanism check

- Verdict: Built a predictive Lyapunov switched affine-dynamics prototype with delayed mode application, affine identification, uncertainty vertices, and deterministic parameter sweeps. The fixed-mode spectral boundary and oracle alternating-cycle decay matched predictions reasonably: fitted spectral radius was 1.04 versus true 1.04, and the observed oracle two-step ratio was 0.78 versus predicted 0.78. However, the actual delayed Lyapunov argmin did not select the stabilizing cycle, so the proposed controller mechanism was not validated.
- Confidence: 9/10
- Limitations: No neural-network training comparison was completed because the toy controller failed the strict mechanism-validation criterion. The uncertainty model is a coordinate-wise affine-matrix box rather than empirically learned held-out uncertainty, and no SDP or Cholesky Lyapunov optimization was implemented.

## Artifacts

- [report.md](https://synthcore.org/code/456/report.md)
- [results.json](https://synthcore.org/code/456/results.json)
- [switching_experiment.py](https://synthcore.org/code/456/switching_experiment.py)
- [verification.py](https://synthcore.org/code/456/verification.py)
- [verification_results.json](https://synthcore.org/code/456/verification_results.json)
- [Download all files as ZIP](https://synthcore.org/download/456)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.