Covariance-Adjusted Training Uncertainty Controller

1. Integration point: wrap ordinary SGD, momentum SGD, or AdamW after gradient clipping and before applying the parameter update. At every minibatch construct a low-dimensional observable vector $y_t$: signed gradient-update alignment $y_1=\langle g_t,\Delta\theta_t\rangle/d$, update traffic $y_2=\|\Delta\theta_t\|_1/d$, gradient traffic $y_3=\|g_t\|_1/d$, and one signed or unsigned update norm for each layer. Here $d$ is the number of sampled coordinates or layer parameters. 2. Pseudocode: append $y_t$ to a FIFO buffer of length $W=100$; compute the sample mean $\widehat J_t$ and covariance $\widehat C_t$; set $\epsilon=10^{-6}\operatorname{tr}(\widehat C_t)/m$ for $m$ observables, with a positive floor; solve $(\widehat C_t+\epsilon I)x=\widehat J_t$; compute $q_t=\widehat J_t^\top x$; if $q_t>q_{\max}$, replace the base learning rate by $\eta_t\sqrt{q_{\max}/q_t}$. Do not explicitly form a matrix inverse. 3. Computed versus estimated: the mean, covariance, linear solve, and learning-rate adjustment are directly computed; the paper’s dissipation interpretation and factor 2 are imported as motivation, not claimed to be an exact SGD entropy-production theorem. Calibrate $q_{\max}$ from a stable warm-up trajectory, such as its 95th percentile. 4. First experiment: train a two-layer MLP on MNIST with SGD and AdamW over a logarithmic learning-rate grid and batch sizes 32, 128, and 512. Compare the controller with no controller and scalar gradient-variance clipping, using identical seeds. Record $q_t$, loss, gradient norms, update norms, and divergence. The prediction is that divergence occurs shortly after a reproducible $q_t=q_c$ crossing, and choosing $q_{\max}$ 5–15% below $q_c$ reduces catastrophic runs by at least 50% with less than 10% additional optimization steps. The measured critical learning rate should shift systematically with batch size and remain more sharply aligned with divergence than any single-observable variance statistic.