# Cycle-Aware Heavy-Ball Safeguard

- ID: 168
- Canonical URL: https://synthcore.org/idea/168/cycle-aware-heavy-ball-safeguard
- API JSON: https://synthcore.org/api/idea/168.json
- API Markdown: https://synthcore.org/api/idea/168.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.25279](https://arxiv.org/abs/2608.25279)
- Category: optimization
- Solves: stability, speedup
- ML areas: optimizer, training-dynamics
- Math tags: dynamical-systems, optimization, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Use the paper's heavy-ball recursion as a runtime diagnostic for momentum optimizers. Detect when recent parameter differences form an approximately periodic orbit or when the estimated local two-step transition matrix has spectral radius near or above one, then reduce the learning rate and momentum temporarily. This targets the failure mode proved in the paper: fixed momentum parameters can produce attracting cycles even on smooth potentials with bounded curvature.

## Mathematical statement

The paper obtains the position recursion x_{j+1}=(1+beta)x_j-beta x_{j-1}-s grad U(x_j), where x_j is the iterate, s>0 is the step size, and beta is the momentum coefficient. Around a periodic point x_j^circ, it assumes the gradient is locally affine, grad U(x_j^circ+u)=grad U(x_j^circ)+H_j u, with symmetric local Hessian H_j and small perturbation u. The resulting local state transition is [u_{j+1};u_j]=A_j[u_j;u_{j-1}], with A_j=[[ (1+beta)I-sH_j, -beta I ], [ I, 0 ]]. For constant curvature H_j=lambda I, stability is controlled by the spectral radius of A(lambda)=[[1+beta-s lambda,-beta],[1,0]]; eigenvalues close to the unit circle imply slow relaxation or oscillatory transients. The paper's explicit cycle satisfies x_{j+1}^circ=(1+beta)x_j^circ-beta x_{j-1}^circ-s grad U(x_j^circ) and demonstrates that a stable periodic attractor can coexist with smooth bounded curvature. The ML adaptation estimates local curvature through Hessian-vector products and estimates periodicity directly from recent optimizer states.

## Key formulas

- $$x_{j+1}=(1+\beta)x_j-\beta x_{j-1}-s\nabla U(x_j)$$
- $$\nabla U(x_j^\circ+u)=\nabla U(x_j^\circ)+H_j u$$
- $$\mathsf A_j=\begin{pmatrix}(1+\beta)I-sH_j&-\beta I\\ I&0\end{pmatrix}$$
- $$\rho\!\left(A(\lambda)\right),\qquad A(\lambda)=\begin{pmatrix}1+\beta-s\lambda&-\beta\\1&0\end{pmatrix}$$

## Implementation notes

Integrate this into the AdamW or SGD-with-momentum update at the parameter-update level, before applying weight decay. Maintain two flattened, random-projected parameter-displacement vectors rather than storing full models: d_t=P(theta_t-theta_{t-1}) and d_{t-1}=P(theta_{t-1}-theta_{t-2}), where P is a fixed Gaussian projection to 32 dimensions. Every 20 optimizer steps, compute the normalized lag-2 correlation c_2=<d_t,d_{t-2}>/(||d_t|| ||d_{t-2}||+epsilon), the displacement ratio r=||d_t||/(||d_{t-1}||+epsilon), and an approximate local curvature interval. Estimate the largest curvature with one or two Hessian-vector products using lambda_hat_max=|v^T H v|/(||v||^2+epsilon), where v is a normalized random parameter vector and Hv is obtained by automatic differentiation; use a gradient-norm or update-norm proxy if Hessian-vector products are unavailable. For current step size s and momentum beta, form the two-by-two matrix A(lambda_hat_max) above and compute its spectral radius exactly. Trigger a safeguard if c_2>0.8, |r-1|<0.15, and either the spectral radius exceeds 0.98 or the same displacement-direction pattern repeats for three windows. On a trigger, set beta to min(0.5,0.7 beta), set s to 0.8s, clear momentum buffers, and hold these values for 100 steps; restore the original values gradually if the trigger disappears. The paper's mathematics supplies the local recurrence and spectral diagnostic; the correlation thresholds, projection dimension, and recovery schedule are empirical. First test on a two-layer MLP and a small Transformer trained on CIFAR-10 or TinyStories, comparing SGD-momentum or AdamW against the safeguard at equal FLOPs. Also test synthetic quadratic losses with eigenvalues logarithmically spaced over condition numbers from 10^2 to 10^6, plus a smooth piecewise-quadratic loss designed to create a period-three orbit. Success means fewer late-training loss plateaus and cycle triggers, lower worst-seed loss, and faster loss descent than fixed momentum. A negative result is no improvement or excessive false-positive damping on well-conditioned tasks.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a projected cycle-aware heavy-ball optimizer diagnostic with exact 2x2 spectral-radius computation and a quadratic stress test. The matrix recurrence matched exactly, and reducing learning rate/momentum lowered the measured spectral radius from approximately 1.000 to 0.815, confirming the local stability diagnostic. However, the safeguard produced much worse divergence than fixed momentum in the tested ill-conditioned quadratic, and the specified lag-2 correlation is -0.5 for an ideal period-three orbit, so it does not detect the paper’s explicit cycle pattern.

### Mechanism check

- Verdict: Built a projected cycle-aware heavy-ball optimizer diagnostic with exact 2x2 spectral-radius computation and a quadratic stress test. The matrix recurrence matched exactly, and reducing learning rate/momentum lowered the measured spectral radius from approximately 1.000 to 0.815, confirming the local stability diagnostic. However, the safeguard produced much worse divergence than fixed momentum in the tested ill-conditioned quadratic, and the specified lag-2 correlation is -0.5 for an ideal period-three orbit, so it does not detect the paper’s explicit cycle pattern.
- Confidence: 9/10
- Limitations: Only a small deterministic numpy quadratic experiment was tested; no MLP, Transformer, CIFAR-10/TinyStories task, AdamW integration, Hessian-vector-product curvature estimate, multi-seed study, FLOP comparison, or smooth piecewise-quadratic attracting cycle was evaluated. The recovery and threshold schedule remains empirical and the tested reduction was insufficient to stabilize this quadratic.

## Artifacts

- [cycle_safeguard.py](https://synthcore.org/code/1194/cycle_safeguard.py)
- [report.md](https://synthcore.org/code/1194/report.md)
- [Download all files as ZIP](https://synthcore.org/download/1194)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.