Muon with Finite Newton-Schulz: The Smoothing Benefit in Nonsmooth Nonconvex Optimization
arXiv:2608.26288
2026
Optimization
1 ideas extracted · analyzed Aug 29, 2026
What the math gives to ML
The paper identifies finite Newton–Schulz depth as a useful smoothing mechanism rather than merely an approximation error. The polar map is discontinuous when singular values cross zero, while a finite polynomial spectral map is Lipschitz in the singular values and can support online-to-nonconvex stationarity arguments. This suggests deliberately replacing exact polar normalization in matrix-valued optimizer updates with a finite, possibly accuracy-dependent Newton–Schulz transform. The most direct test is an adaptive-depth Muon variant that uses shallow iterations early and increases depth only as the target stationarity or training accuracy becomes stricter.
Ideas from this paper
✓ Mechanism works
Re-invented
2026
Use a finite Newton–Schulz spectral transform on Muon momentum matrices instead of computing an exact polar factor or treating finite depth as a nuisance approximation. The finite polynomial remains close to orthogonalization but smooths the singular-value response, reducing abrupt update changes caused by rank deficiency or small singular values. Increase the iteration depth logarithmically with training progress or a target error rather than using a fixed expensive depth.
Useful8/10
Difficulty4/10
Novelty4/10