Forced Variational Momentum Optimizer

1. Integration point: implement this as a new PyTorch optimizer that replaces the parameter-update line of SGD or heavy-ball. Maintain one previous parameter tensor for every trainable tensor. Use a positive diagonal mass tensor M; initialize M to ones for the first experiment, and later test a frozen or slowly varying diagonal preconditioner. 2. Pseudocode: initialize θ_prev = θ and compute g = gradient of the current minibatch loss. Set a = γh/2, rho = (1-a)/(1+a), and step = h²/(1+a). Compute velocity = θ - θ_prev. Update θ_new = θ + rho * velocity - step * g / M elementwise. Then assign θ_prev = θ and θ = θ_new. Initialize θ_prev equal to θ so that the first update has zero momentum. Apply decoupled weight decay after this update if required. 3. Computed from the paper's mathematics: rho and step follow directly from the discrete Lagrangian and symmetric force split. Estimated empirically: sweep h and γ, and optionally set M_i = sqrt(EMA(g_i²)) + ε after a warmup, while preserving the same update structure. 4. First experiment: train a two-layer MLP on MNIST or CIFAR-10 using batch size 256. Compare against SGD, heavy-ball momentum, and AdamW at equal gradient evaluations. Measure loss versus steps, maximum stable h, gradient-norm spikes, parameter-update norms, validation accuracy, and optimizer-state memory. A positive result is faster loss descent or a wider stable step-size range than heavy-ball, with fewer oscillatory loss spikes and substantially less state memory than AdamW.