Cycle-Aware Heavy-Ball Safeguard

Integrate this into the AdamW or SGD-with-momentum update at the parameter-update level, before applying weight decay. Maintain two flattened, random-projected parameter-displacement vectors rather than storing full models: d_t=P(theta_t-theta_{t-1}) and d_{t-1}=P(theta_{t-1}-theta_{t-2}), where P is a fixed Gaussian projection to 32 dimensions. Every 20 optimizer steps, compute the normalized lag-2 correlation c_2=<d_t,d_{t-2}>/(||d_t|| ||d_{t-2}||+epsilon), the displacement ratio r=||d_t||/(||d_{t-1}||+epsilon), and an approximate local curvature interval. Estimate the largest curvature with one or two Hessian-vector products using lambda_hat_max=|v^T H v|/(||v||^2+epsilon), where v is a normalized random parameter vector and Hv is obtained by automatic differentiation; use a gradient-norm or update-norm proxy if Hessian-vector products are unavailable. For current step size s and momentum beta, form the two-by-two matrix A(lambda_hat_max) above and compute its spectral radius exactly. Trigger a safeguard if c_2>0.8, |r-1|<0.15, and either the spectral radius exceeds 0.98 or the same displacement-direction pattern repeats for three windows. On a trigger, set beta to min(0.5,0.7 beta), set s to 0.8s, clear momentum buffers, and hold these values for 100 steps; restore the original values gradually if the trigger disappears. The paper's mathematics supplies the local recurrence and spectral diagnostic; the correlation thresholds, projection dimension, and recovery schedule are empirical. First test on a two-layer MLP and a small Transformer trained on CIFAR-10 or TinyStories, comparing SGD-momentum or AdamW against the safeguard at equal FLOPs. Also test synthetic quadratic losses with eigenvalues logarithmically spaced over condition numbers from 10^2 to 10^6, plus a smooth piecewise-quadratic loss designed to create a period-three orbit. Success means fewer late-training loss plateaus and cycle triggers, lower worst-seed loss, and faster loss descent than fixed momentum. A negative result is no improvement or excessive false-positive damping on well-conditioned tasks.