Stochastic Gradient Descent with Momentum: Analysis and Synthesis via Integral Quadratic Constraints
arXiv:2608.06915
2026
Optimization
1 ideas extracted · analyzed Aug 31, 2026
What the math gives to ML
The paper provides a control-theoretic route for designing stochastic first-order optimizers rather than selecting momentum and step-size parameters by hand. Its transferable asset is the dynamic integral quadratic constraint (IQC): temporal filters encode second-order information about mini-batch gradient noise and curvature, while a semidefinite program searches for a controller and Lyapunov certificate with a small certified contraction factor. A practical neural-network adaptation is to synthesize a low-order momentum filter offline on representative training trajectories, then deploy the resulting recurrence as a drop-in optimizer and compare it against tuned SGD, Heavy Ball, Nesterov, and AdamW.
Ideas from this paper
✗ Failed on benchmark
2026
Replace hand-designed Heavy Ball or Nesterov coefficients with a low-order linear controller synthesized by a semidefinite program. The controller receives the stochastic mini-batch gradient and emits the parameter update; dynamic IQC multipliers constrain both gradient curvature and temporally correlated mini-batch noise, so the SDP directly minimizes a certified contraction factor rather than optimizing momentum heuristically.
Useful7/10
Difficulty7/10
Novelty7/10