# Forced Variational Momentum Optimizer

- ID: 372
- Canonical URL: https://synthcore.org/idea/372/forced-variational-momentum-optimizer
- API JSON: https://synthcore.org/api/idea/372.json
- API Markdown: https://synthcore.org/api/idea/372.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.02694](https://arxiv.org/abs/2607.02694)
- Category: optimization
- Solves: stability, speedup, generalization
- ML areas: optimizer, training-dynamics
- Math tags: symplectic, hamiltonian, dynamical-systems, numerical-analysis, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Replace standard heavy-ball momentum with an update derived from a discrete kinetic-minus-loss action and a discrete viscous force. The force discretization produces a rational damping factor that remains controlled over a specified range of step sizes, potentially reducing oscillations and instability without Adam-style second-moment state.

## Mathematical statement

The central result is the forced discrete Euler–Lagrange equation $$D_{2}L_{d}(q_{k-1},q_{k})+D_{1}L_{d}(q_{k},q_{k+1})+f_{d}^{+}(q_{k-1},q_{k})+f_{d}^{-}(q_{k},q_{k+1})=0.$$ Here Q is the configuration space, q_k is the state at step k, L_d: Q × Q → R is a discrete Lagrangian, D_1 and D_2 are derivatives with respect to its first and second arguments, and f_d = f_d^- + f_d^+ is a discrete force split into covectors acting on the two endpoints. For neural-network parameters θ, choose $$L_d(θ_k,θ_{k+1})=\frac{1}{2h}(θ_{k+1}-θ_k)^\top M(θ_{k+1}-θ_k)-hF(θ_k),$$ where F is the training loss, h > 0 is the step size, and M is a positive diagonal mass matrix. Approximate viscous forcing by $$f_d^-(θ_k,θ_{k+1})=f_d^+(θ_k,θ_{k+1})=-\frac{γ}{2}M(θ_{k+1}-θ_k),$$ where γ ≥ 0 is damping. Substitution gives $$θ_{k+1}=θ_k+\frac{1-γh/2}{1+γh/2}(θ_k-θ_{k-1})-\frac{h^2}{1+γh/2}M^{-1}∇F(θ_k).$$ The bounded rational coefficient is the key transferable property.

## Key formulas

- $$D_{2}L_{d}(q_{k-1},q_{k})+D_{1}L_{d}(q_{k},q_{k+1})+f_{d}^{+}(q_{k-1},q_{k})+f_{d}^{-}(q_{k},q_{k+1})=0.$$
- $$L_d(θ_k,θ_{k+1})=\frac{1}{2h}(θ_{k+1}-θ_k)^\top M(θ_{k+1}-θ_k)-hF(θ_k),$$
- $$f_d^-(θ_k,θ_{k+1})=f_d^+(θ_k,θ_{k+1})=-\frac{γ}{2}M(θ_{k+1}-θ_k).$$
- $$θ_{k+1}=θ_k+\frac{1-γh/2}{1+γh/2}(θ_k-θ_{k-1})-\frac{h^2}{1+γh/2}M^{-1}∇F(θ_k).$$

## Implementation notes

1. Integration point: implement this as a new PyTorch optimizer that replaces the parameter-update line of SGD or heavy-ball. Maintain one previous parameter tensor for every trainable tensor. Use a positive diagonal mass tensor M; initialize M to ones for the first experiment, and later test a frozen or slowly varying diagonal preconditioner. 2. Pseudocode: initialize θ_prev = θ and compute g = gradient of the current minibatch loss. Set a = γh/2, rho = (1-a)/(1+a), and step = h²/(1+a). Compute velocity = θ - θ_prev. Update θ_new = θ + rho * velocity - step * g / M elementwise. Then assign θ_prev = θ and θ = θ_new. Initialize θ_prev equal to θ so that the first update has zero momentum. Apply decoupled weight decay after this update if required. 3. Computed from the paper's mathematics: rho and step follow directly from the discrete Lagrangian and symmetric force split. Estimated empirically: sweep h and γ, and optionally set M_i = sqrt(EMA(g_i²)) + ε after a warmup, while preserving the same update structure. 4. First experiment: train a two-layer MLP on MNIST or CIFAR-10 using batch size 256. Compare against SGD, heavy-ball momentum, and AdamW at equal gradient evaluations. Measure loss versus steps, maximum stable h, gradient-norm spikes, parameter-update norms, validation accuracy, and optimizer-state memory. A positive result is faster loss descent or a wider stable step-size range than heavy-ball, with fewer oscillatory loss spikes and substantially less state memory than AdamW.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a PyTorch ForcedVariationalMomentum optimizer and reproducible verification/benchmark script. The discrete Euler–Lagrange recurrence matched the implemented update to numerical precision, and the quadratic sweep showed a small wider stable h range than the selected heavy-ball control (0.6266 vs 0.6067) with bounded rational damping. However, on the fixed digits MLP task the idea had worse test loss and accuracy than both controls (0.2107/96.0% vs heavy-ball 0.1119/96.9%), so the claimed practical optimization win was not observed despite fewer large loss spikes and AdamW's doubled state memory.

### Mechanism check

- Verdict: Built a PyTorch ForcedVariationalMomentum optimizer and reproducible verification/benchmark script. The discrete Euler–Lagrange recurrence matched the implemented update to numerical precision, and the quadratic sweep showed a small wider stable h range than the selected heavy-ball control (0.6266 vs 0.6067) with bounded rational damping. However, on the fixed digits MLP task the idea had worse test loss and accuracy than both controls (0.2107/96.0% vs heavy-ball 0.1119/96.9%), so the claimed practical optimization win was not observed despite fewer large loss spikes and AdamW's doubled state memory.
- Confidence: 9/10
- Limitations: Only one small digits dataset, one primary hyperparameter setting per optimizer, and 450 minibatch updates were tested. The mass was scalar one rather than a learned or frozen diagonal preconditioner; no broad hyperparameter search, FLOP-normalized timing, weight decay, or multiple random seeds was evaluated.

## Artifacts

- [experiment.py](https://synthcore.org/code/1278/experiment.py)
- [report.md](https://synthcore.org/code/1278/report.md)
- [results.json](https://synthcore.org/code/1278/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1278)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.