Schur-Stable Second-Order Optimizer

Integrate this at the optimizer-update level, maintaining one velocity tensor \(v\) for every trainable parameter tensor \(\theta\). Use \(\Delta t=1\), compute the minibatch gradient \(g\), and update with the paper-inspired semi-implicit ordering: first update velocity, then parameters. A concrete implementation is: (1) initialize \(v=0\), choose damping \(\gamma\in[0.2,1.5]\), and set a safety factor \(s\in[0.5,0.9]\); (2) every \(K\) optimizer steps, estimate a global curvature proxy \(h_{\max}\) using one or two Hessian-vector products with a random vector, \(h_{\max}\approx\|Hq\|/\|q\|\), optionally taking an exponential moving maximum across layers; (3) set \(\eta=s(4-2\gamma)/(h_{\max}+\delta)\), clipped to a user-specified upper bound, so the estimated largest mode satisfies the Schur condition; (4) compute \(v\leftarrow(1-\gamma)v-\eta g\), then \(\theta\leftarrow\theta+v\); and (5) optionally apply decoupled weight decay after the state update. For noisy minibatch training, estimate the error floor empirically from a moving average of loss variance and stop increasing \(\eta\) when the loss reaches a stable plateau. The stability region and geometric-decay expression come from the paper; the Hessian estimate, safety factor, and layerwise aggregation are engineering choices. First test on a 6-layer transformer or 3-layer MLP on CIFAR-10, comparing this optimizer with momentum SGD at matched wall-clock and AdamW at matched parameter-update count. Measure training loss per FLOP, gradient-norm explosions, learning-rate oscillations, and final validation accuracy. A useful success signal is stable training at an effective step size at least 1.5 times the momentum-SGD baseline, with faster loss reduction at equal FLOPs and no increase in validation loss.