# Smooth Spectral Muon

- ID: 1689
- Canonical URL: https://synthcore.org/idea/1689/smooth-spectral-muon
- API JSON: https://synthcore.org/api/idea/1689.json
- API Markdown: https://synthcore.org/api/idea/1689.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.01911](https://arxiv.org/abs/2608.01911)
- Category: optimization
- Solves: stability, speedup, generalization
- ML areas: optimizer, transformer, mlp, training-dynamics
- Math tags: spectral-theory, linear-algebra, optimization, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 4/10

## Idea description

Replace the exact matrix-polar normalization in Muon with the smoothed feedback \(h_\epsilon(M)=M(M^\top M+\epsilon I)^{-1/2}\). This retains singular-vector-aware updates and approximately unit-normalizes dominant spectral modes, but avoids unstable behavior when the momentum matrix is rank deficient or has tiny singular values.

## Mathematical statement

The paper defines the smoothed matrix-polar feedback for a matrix \(Z\in\mathbb{R}^{m\times n}\) and \(\epsilon>0\) as \(h_\epsilon(Z)=Z(Z^\top Z+\epsilon I)^{-1/2}\). If \(Z=U\operatorname{diag}(\sigma_i)V^\top\) is an SVD, then \(h_\epsilon(Z)=U\operatorname{diag}(\sigma_i/\sqrt{\sigma_i^2+\epsilon})V^\top\), so each singular value is transformed by \(r_\epsilon(\sigma)=\sigma/\sqrt{\sigma^2+\epsilon}\): it is approximately \(1\) for \(\sigma\gg\sqrt{\epsilon}\) and approximately \(\sigma/\sqrt{\epsilon}\) near zero. The response is bounded by \(0\le r_\epsilon(\sigma)<1\), preventing arbitrarily large normalized components. The direction is the gradient of the spectral energy \(\Phi_\epsilon(Z)=\operatorname{Tr}[(Z^\top Z+\epsilon I)^{1/2}-\sqrt{\epsilon}I]\), namely \(\nabla\Phi_\epsilon(Z)=h_\epsilon(Z)\). In the optimizer, \(Z=M_k\) is the momentum matrix, \(M_k\) is the exponentially averaged gradient, \(\eta\) is the learning rate, and \(\beta\) is momentum.

## Key formulas

- $$M_k=\beta M_{k-1}+\nabla f(W_k),\qquad W_{k+1}=W_k-\eta\,\operatorname{Polar}(M_k),\qquad \operatorname{Polar}(M)=M(M^\top M)^{-1/2}.$$
- $$h_\epsilon(Z)=Z(Z^\top Z+\epsilon I)^{-1/2}.$$
- $$Z=U\operatorname{diag}(\sigma_i)V^\top\quad\Longrightarrow\quad h_\epsilon(Z)=U\operatorname{diag}\left(\frac{\sigma_i}{\sqrt{\sigma_i^2+\epsilon}}\right)V^\top.$$
- $$\Phi_\epsilon(Z)=\operatorname{Tr}\left[(Z^\top Z+\epsilon I)^{1/2}-\sqrt{\epsilon}I\right],\qquad \nabla\Phi_\epsilon(Z)=h_\epsilon(Z).$$

## Implementation notes

Integrate this into the optimizer update for every trainable two-dimensional weight matrix, especially transformer projection matrices and MLP weights; leave embeddings, biases, and layer-normalization vectors on AdamW or ordinary SGD. Maintain one momentum matrix per selected weight. At each training step, compute the gradient G, update M <- beta*M + G, and replace the exact polar update by a smoothed spectral update. A numerically robust implementation is: (1) compute the thin SVD M=U diag(s) V^T; (2) form s_scaled=s/sqrt(s*s+epsilon); (3) D=U diag(s_scaled) V^T; (4) W <- W - eta*D. For large matrices, benchmark this against Newton-Schulz polar iteration, but use the exact SVD version first as the correctness reference. Epsilon is a scalar or per-layer value; start with epsilon=c times the squared RMS singular value, where RMS_s=(mean_i s_i^2)^(1/2) and c is in {1e-4,1e-3,1e-2}, then optionally EMA-smooth RMS_s across steps. The paper-derived computation is the singular response s_scaled=s/sqrt(s^2+epsilon) and the potential Phi_epsilon; empirical choices are beta, eta, c, and whether epsilon is fixed or adaptive. Log the minimum singular value, maximum update singular value, update Frobenius norm, and Phi_epsilon(M). First test on a 12-layer small Transformer language model such as GPT-2-small or a 100M-parameter decoder on WikiText-103, comparing AdamW, exact Muon, and smoothed Muon at matched optimizer FLOPs. The initial success signal is fewer NaNs and less update-norm variance on deliberately small batches or low-rank gradient settings, followed by lower validation perplexity or faster training-loss reduction at equal steps. Also test epsilon ablations to verify that large epsilon improves stability but excessive epsilon harms convergence by making the update too linear.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built and numerically verified Smooth Spectral Muon with a thin-SVD implementation, spectral potential, finite-difference gradient check, and toy low-rank matrix regression. The predicted transition and asymptotic scaling matched essentially exactly: transition at sigma/sqrt(epsilon)=1, small-mode ratio 0.999999995, large-mode asymptotic 1.000000016, bounded response below 1, and matrix spectral error 0. The smoothed optimizer retained Muon-like loss while providing finite update variability, but did not beat SGD, so this confirms the mechanism rather than an overall training win.

### Mechanism check

- Verdict: Built and numerically verified Smooth Spectral Muon with a thin-SVD implementation, spectral potential, finite-difference gradient check, and toy low-rank matrix regression. The predicted transition and asymptotic scaling matched essentially exactly: transition at sigma/sqrt(epsilon)=1, small-mode ratio 0.999999995, large-mode asymptotic 1.000000016, bounded response below 1, and matrix spectral error 0. The smoothed optimizer retained Muon-like loss while providing finite update variability, but did not beat SGD, so this confirms the mechanism rather than an overall training win.
- Confidence: 9/10
- Limitations: Only a small NumPy low-rank matrix-regression toy was tested. No Transformer, language modeling, WikiText, AdamW, CUDA benchmark, matched-FLOP study, NaN stress test, or multi-seed statistical evaluation was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/623/bench_report.json)
- [report.md](https://synthcore.org/code/623/report.md)
- [report_bench_2026-08-31T200515.md](https://synthcore.org/code/623/report_bench_2026-08-31T200515.md)
- [results.json](https://synthcore.org/code/623/results.json)
- [smooth_spectral_muon.py](https://synthcore.org/code/623/smooth_spectral_muon.py)
- [stage2_bench.py](https://synthcore.org/code/623/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/623)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.