Predictive Lyapunov Switched Optimizer

Integration point: insert the controller immediately before the optimizer step in an ordinary PyTorch training loop. Define three to five modes with identical model parameters but different update rules or learning rates: conservative SGD with learning rate $\eta$, momentum SGD with $2\eta$, Adam with $\eta$, and a damping mode with $\eta/4$. Maintain a compact state $z_k$ rather than a full parameter-space model: concatenate current loss, log gradient norm, cosine similarity between successive gradients, normalized update norm, and a random projection of the parameter vector. First collect trajectories under each mode separately and fit $\bar A_j,\bar b_j$ by ridge regression from $(z_k,z_{k+1})$. Estimate uncertainty by fitting residual ellipsoids or coordinate-wise boxes on held-out trajectories; use their vertices as $\mathcal V_j$. Choose a two- or three-phase cycle by clustering recurring states, or initialize a two-phase cycle from low-loss and recovery states. Solve for diagonal or full $P_i\succ0$ using a small semidefinite program over sampled uncertainty vertices, enforcing the displayed decrease inequality. If an SDP is inconvenient, optimize Cholesky factors of $P_i$ with hinge penalties. Because the controller is delayed, queue the selected mode and apply the previously queued mode at the next step. At each iteration, observe $z_k$, predict $\hat z_{k+1}^{(j)}$ for every mode, compute all vertex Lyapunov differences, select the mode with the smallest worst-case score, queue it, apply the previous queued mode, update the network, and log $V_i$ and certificate violations. The paper supplies the phase-dependent Lyapunov and robust-vertex logic; the neural-network state, affine models, uncertainty sets, cycle centers, and optimizer modes are estimated empirically. The first cheap experiment should use a two-layer MLP on MNIST, comparing Adam, SGD, cosine-scheduled SGD, and the switched controller at matched average update cost. Deliberately test learning rates around the fixed-mode instability boundary. The quantitative prediction is that a fixed mode becomes unstable when its local predicted closed-loop spectral radius crosses one, approximately $\rho(\bar A_j)\approx1$. With switching active, phase-conditioned Lyapunov drift should be negative outside a residual uncertainty set, and $\|z_k-\rho_{i_k}\|$ should decay geometrically until reaching the uncertainty floor. Test whether the empirical divergence threshold is predicted within 20% by the fitted spectral radius and whether switched training remains bounded beyond the best fixed-mode threshold.