Singular-gap controlled stochastic optimizer

(1) Integrate this into the optimizer hyperparameter controller, not the forward network. Use a stochastic training state $z_k=(\theta_k,m_k)$, where $m_k$ is momentum or Adam state; the optimizer update plus minibatch sampling defines the transition $z_{k+1}\sim P(z_k,\cdot)$. Every $M$ steps, record short windows of $z_k$ or low-dimensional observables $h_j(z_k)$ such as loss, gradient norm, momentum norm, and random parameter projections. Center each observable within the window: $\tilde h_j=h_j-\frac1T\sum_{k=0}^{T-1}h_j(z_k)$. (2) Estimate the action of $P$ by fitting $h_j(z_{k+1})\approx\sum_i A_{ij}h_i(z_k)$ with ridge regression on centered probe vectors. Estimate the probe covariance $C$, whiten with $C^{-1/2}$, and form $A_w=C^{-1/2}AC^{1/2}$; compute $\hat s=\sigma_{\max}(A_w)$ using a small SVD. For a rotational optimizer, compute $\hat A_{avg}=K^{-1}\sum_{r=0}^{K-1}\hat A^r$. (3) The paper-derived quantities are the centered invariant-space restriction, the $P^*P$ singular-value criterion, and the time-averaged operator. The regression matrix, covariance, and finite-dimensional singular value are empirical approximations. If the EMA of $\hat s$ exceeds 0.98, reduce momentum $\beta$ or learning rate $\eta$ by 10%; if it is below 0.90 while loss decreases, increase $\eta$ by 5%. Update only every 500 steps and use a 0.9 EMA to avoid noisy control. (4) First test on a two-layer MLP and small ResNet on MNIST or CIFAR-10, comparing fixed tuned SGD-momentum and AdamW against the controller. Measure loss and validation accuracy at equal steps and wall-clock, gradient spikes, oscillation of loss, and $\hat s$. The hypothesis is that the singular gap predicts transient instability better than eigenvalue-style autocorrelation, yielding at least 10% lower loss after 5,000 steps or 20% fewer steps to a target accuracy without material probe overhead.