arXiv:2608.22994 2026 Optimization 1 ideas extracted · analyzed Sep 1, 2026
The paper supplies a physically motivated replacement for Muon's single exponential momentum: represent optimizer memory as multiple internal stress modes, each obeying its own first-order relaxation law. The key transferable asset is the explicit separation of fast and slow gradient memory, combined linearly before Muon's matrix-geometric update, rather than tuning one momentum coefficient to compromise between early rapidly changing gradients and later stable gradients. This yields a simple optimizer variant with only two extra matrix states and a small number of scalar hyperparameters, while preserving Muon's semi-orthogonalization. The most direct test is to replace Muon's momentum buffer in a small transformer and compare loss-versus-steps and loss-versus-FLOPs against Muon and AdamW.
Replace Muon's single momentum matrix with a weighted mixture of fast and slow relaxation modes. The fast mode tracks rapidly changing gradients while the slow mode preserves a longer-horizon direction; their mixture is semi-orthogonalized and applied as the matrix update.