A Unified Theory of H-Duality in First-Order Methods
arXiv:2609.03281
2026
Optimization
1 ideas extracted · analyzed Sep 4, 2026
What the math gives to ML
The paper identifies H-duality between a fixed-step first-order method and its time-reversed update sequence: a schedule that is strong for terminal function-value error can have a reversed counterpart that is strong for terminal gradient or oracle-residual error. The transferable asset is a finite-horizon schedule transformation backed by exact linear recurrences on extremal-curvature quadratic models. A practical neural-network adaptation is to optimize short momentum schedules on a small family of quadratic curvature proxies, deploy both the learned and reversed schedules, and test whether one gives lower terminal gradient norms or faster loss descent at equal optimizer cost.
Ideas from this paper
Unverified
2026
Construct a finite-horizon momentum schedule and its coefficient-reversed counterpart, then use the reversed schedule when the desired endpoint criterion is gradient norm rather than function-value decrease. Optimize the schedule on extremal-curvature quadratic proxies, where the dynamics reduce to an exactly simulable scalar recurrence; this creates a cheap, falsifiable optimizer-design procedure without changing the neural-network architecture.
Useful6/10
Difficulty5/10
Novelty6/10