Mirror Polyak and a Primal-Dual Lifting
arXiv:2608.17252
2026
Optimization
1 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper develops a Polyak-style step rule for mirror descent, replacing Euclidean gradient geometry with a problem-specific Bregman geometry. The transferable asset is objective-gap adaptation combined with mirror maps that encode positivity, simplex constraints, entropy geometry, or other non-Euclidean structure. A practical neural-network experiment is to use this rule for constrained parameters such as mixture weights, MoE router probabilities, or positive gates, solving a one-dimensional scalar equation for each update. Direct use in fully nonconvex stochastic training should begin with conservative lower-bound estimates and step clipping.
Ideas from this paper
✗ Failed on benchmark
2026
Replace the Euclidean Polyak step in an optimizer with a mirror-descent step whose length is chosen by projecting onto the current affine lower-bound halfspace in Bregman geometry. This permits entropy geometry for simplex-valued router probabilities, log geometry for positive parameters, and other mirror maps without reducing the method to a norm-based learning-rate rule.
Useful7/10
Difficulty5/10
Novelty7/10