arXiv:2607.03519 2026 Optimization 1 ideas extracted · analyzed Aug 30, 2026
The paper gives a constructive adversarial sequence showing that projected Adam can have nonvanishing average regret for every choice of moment-decay parameters, removing the earlier restriction \(\beta_1<\sqrt{\beta_2}\). Its transferable asset is not a new optimizer, but a minimal periodic gradient process that exposes destructive interaction between momentum, second-moment normalization, and projection; the same mechanism applies to AdamW, RMSProp, NAdam, Adan, AdaMax, and Muon. The most practical use is an optimizer stress-test and a runtime guard that detects this gradient-ordering pathology and temporarily switches to a non-adaptive update.
Use the paper's three-periodic linear-gradient construction as a cheap online detector for Adam's failure mode. When recent gradients exhibit the pattern \((c,-1,-1)\), with \(c>2\), and the adaptive update repeatedly moves in a harmful direction, freeze Adam's normalization and use a short SGD or AMSGrad fallback before returning to Adam.