Saturating low-rank coupled optimizer
Source paper: Spectral gap for the three-dimensional damped cubic wave equation with degenerate noise arXiv:2608.28459 ⓘ · analyzed Sep 2, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Train two parameter replicas with common low-rank stochastic forcing and an adaptive finite-dimensional Cameron–Martin correction that contracts their discrepancy in a weak parameter metric. Transporting the forcing directions through the loss Hessian is intended to make a rank-k perturbation influence more than k raw parameter directions, while damped momentum suppresses high-energy divergence.
Formulas
Mathematical statement
The source equation is the damped second-order stochastic system $\mathrm{d}u=v\,\mathrm{d}t$, $\mathrm{d}v=(\Delta u-a(x)v-u^{3})\,\mathrm{d}t+B\,\mathrm{d}W_t$, where $u$ is a field, $v$ is its velocity, $a(x)>0$ is damping, $B$ maps finite-dimensional Brownian motion into the state space, and $W_t$ is Brownian motion. The paper measures phase discrepancies in the weak negative topology $\mathcal{E}_s=H^{-s}(\mathbb{T}^3)\times H^{-1-s}(\mathbb{T}^3)$ for $0<s<1/2$, rather than the strong energy topology. Its stable–compact asymptotic coupling principle separates contraction into a compact defect in the weak topology plus a finite-dimensional Cameron–Martin shift, with dissipation handling high-energy states. The ML adaptation uses parameters and momentum $(\theta,p)$, injects noise through $B\in\mathbb{R}^{d\times k}$ with $k\ll d$, and estimates nonlinear saturation using the empirical span of $B,HB,H^2B,\ldots$, where $H$ is a Hessian-vector-product approximation. Define $Q_r=[B,HB,\ldots,H^{r-1}B]$ and $\rho_r=\operatorname{rank}_\varepsilon(Q_r)/d$, where $\operatorname{rank}_\varepsilon$ counts singular values above tolerance $\varepsilon$.
Implementation notes
(1) Integration point: replace the momentum update in SGD with a two-replica optimizer. Maintain $(\theta^A,p^A)$ and $(\theta^B,p^B)$, use the same minibatches and Gaussian vector $\xi_t\in\mathbb{R}^k$ for both replicas, and inject $B\xi_t$ only into momentum. Choose $B$ as k normalized random parameter directions or low-rank directions from recent gradient covariance. Define a weak discrepancy $D_w=\|P_m(\theta^A-\theta^B)\|_2+\alpha\|P_m(p^A-p^B)\|_2+\beta\|(I-P_m)(\theta^A-\theta^B)\|_2/\sqrt{m}$, where $P_m$ retains low-frequency or randomly selected parameter blocks and therefore downweights fine-scale discrepancies. (2) Pseudocode: compute paired gradients $g_A,g_B$; update $p^i\leftarrow(1-\gamma\eta)p^i-\eta g_i+\sigma\sqrt{\eta}B\xi$ and $\theta^i\leftarrow\theta^i+\eta p^i$. Every T steps, compute Hessian-vector products at $\bar\theta=(\theta^A+\theta^B)/2$, append $B,HB,\ldots,H^{r-1}B$ to $Q_r$, and compute a thin SVD. If $D_w$ is increasing, apply $c=Q_r(Q_r^TQ_r+\lambda I)^{-1}(\theta^B-\theta^A)$ and set $\theta^B\leftarrow\theta^B+\kappa c$, with $\kappa$ clipped to $[-\kappa_{max},\kappa_{max}]$. (3) Compute from the paper's mechanism: damped second-order dynamics, finite-rank forcing, weak metric, and finite-dimensional correction. Estimate empirically: Hessian-vector products, $\rho_r$, correction coefficients, and contraction rates; never form H explicitly. (4) First experiment: train a two-layer width-512 MLP on MNIST or CIFAR-10 against SGD with momentum, AdamW, and isotropic underdamped Langevin noise. Match parameter count, optimizer steps, batch size, and total HVP FLOPs. Sweep $k\in\{1,2,4,8,16\}$ and $r\in\{1,2,4,8\}$. (5) Pre-register predictions: improvement should appear when $\rho_r$ exceeds 0.5, and the slope of $\log D_w$ over 200-step windows should correlate negatively with $\rho_r$ (Spearman correlation below -0.5). Correction should help mostly below the 90th percentile of $\|p\|^2+\|g\|^2$, while damping should dominate above it. Require 20% faster loss decrease at equal FLOPs or 10% lower seed variance. Ablate only transport by replacing $HB,H^2B$ with fresh random directions while preserving rank, noise variance, damping, and correction frequency. Falsify the transfer if transported directions do not outperform fresh random directions, if $\rho_r$ fails to predict contraction, or if gains vanish after HVP costs are equalized.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.