# Lipschitz Finite-Newton–Schulz Muon

- ID: 194
- Canonical URL: https://synthcore.org/idea/194/lipschitz-finite-newton-schulz-muon
- API JSON: https://synthcore.org/api/idea/194.json
- API Markdown: https://synthcore.org/api/idea/194.md
- Verification status: mechanism_works
- Source: [arXiv:2608.26288](https://arxiv.org/abs/2608.26288)
- Category: optimization
- Solves: stability, accuracy, speedup
- ML areas: optimizer, training-dynamics, transformer
- Math tags: optimization, linear-algebra, spectral-theory, approximation-theory
- Ratings: usefulness 8/10; difficulty 4/10; novelty 4/10

## Idea description

Use a finite Newton–Schulz spectral transform on Muon momentum matrices instead of computing an exact polar factor or treating finite depth as a nuisance approximation. The finite polynomial remains close to orthogonalization but smooths the singular-value response, reducing abrupt update changes caused by rank deficiency or small singular values. Increase the iteration depth logarithmically with training progress or a target error rather than using a fixed expensive depth.

## Mathematical statement

For a matrix S in R^{m x n} with thin SVD S = U diag(sigma_1(S),...,sigma_r(S)) V^T, the paper defines polar(S) = U diag(1{sigma_i(S)>0}) V^T and states the operator/nuclear duality identity sup_{||X||_op <= D} <S,X> = D ||S||_*; the maximizer is D polar(S). More generally, for h:[0,infinity) -> R with h(0)=0, the spectral map H_h(S) = U diag(h(sigma_1(S)),...,h(sigma_r(S))) V^T. The transfer uses the finite Newton–Schulz polynomial h_t. First scale S by alpha >= ||S||_op, set X_0 = S/alpha, and iterate X_{j+1} = 0.5 X_j(3I - X_j^T X_j) for j=0,...,t-1 when m >= n; for n > m use X_{j+1}=0.5(3I-X_j X_j^T)X_j. On every singular direction, the scalar recursion is h_{j+1}(s)=0.5 h_j(s)(3-h_j(s)^2), with h_0(s)=s/alpha. As t grows, h_t(s) approaches 1 for nonzero normalized singular values, recovering the polar factor, but finite t is a smooth polynomial response. Use G_t = alpha X_t as the optimizer direction. The paper's central transferable claim is that finite depth smooths the discontinuous polar spectral map and that depth O(log(1/epsilon)) is sufficient for an epsilon-accurate nonsmooth nonconvex stationarity guarantee, whereas exact polar updates can fail under the same conversion.

## Key formulas

- $$\sup_{\lVert X\rVert_{\mathrm{op}}\leq D}\langle S,X\rangle=D\lVert S\rVert_{*},\qquad \operatorname*{argmax}_{\lVert X\rVert_{\mathrm{op}}\leq D}\langle S,X\rangle=D\,\operatorname{polar}(S)$$
- $$\mathcal{H}_{h}(S)=U\operatorname{diag}\bigl(h(\sigma_{1}(S)),\ldots,h(\sigma_{r}(S))\bigr)V^{\top},\qquad \operatorname{polar}(S)=U\operatorname{diag}(\mathbf{1}\{\sigma_i(S)>0\})V^{\top}$$
- $$X_0=\frac{S}{\alpha},\quad \alpha\geq\lVert S\rVert_{\mathrm{op}},\qquad X_{j+1}=\frac{1}{2}X_j(3I-X_j^{\top}X_j),\qquad G_t=\alpha X_t$$
- $$h_{j+1}(s)=\frac{1}{2}h_j(s)\bigl(3-h_j(s)^2\bigr),\qquad h_0(s)=\frac{s}{\alpha},\qquad t=O\!\left(\log\frac{1}{\varepsilon}\right)$$

## Implementation notes

(1) Integrate at the Muon optimizer update for every 2-D weight matrix W, after momentum or gradient accumulation and before applying the parameter step. Let M_t = beta M_{t-1} + (1-beta) grad_t, where beta is the momentum coefficient. Replace exact SVD-based polar(M_t), or the existing fixed-depth orthogonalization, by finite Newton–Schulz. This changes only the direction calculation; Adam-style scalar updates for biases and LayerNorm parameters remain unchanged.

(2) Pseudocode:
```text
for each matrix parameter W with gradient G:
    M = beta*M + (1-beta)*G
    alpha = max(eps, estimate_operator_norm(M))
    X = M / alpha
    if rows(M) >= cols(M):
        repeat j = 1..t:
            X = 0.5 * X @ (3*I - X.T @ X)
    else:
        repeat j = 1..t:
            X = 0.5 * (3*I - X @ X.T) @ X
    direction = alpha * X
    W = W - lr * shape_scale(W, direction) * direction
```
Use power iteration with 2–3 matrix-vector products to estimate alpha = ||M||_op, and clamp alpha to at least 1e-6. Start with t=2 or 3. For an adaptive schedule, choose t_t = min(t_max, max(t_min, ceil(c log(1+q_t)))) where q_t is the normalized training-step fraction or a running inverse gradient-noise estimate; a simpler first test uses t=2 for the first 70% of training and t=4 thereafter.

(3) The paper supplies the spectral-map construction, polar/nuclear duality, and the logarithmic-depth smoothing guarantee. Estimate ||M||_op empirically by power iteration, and measure the actual orthogonalization residual r_t = ||X_t^T X_t-I||_F / sqrt(min(m,n)). Also estimate update smoothness by recording ||direction_t-direction_{t-1}||_F / ||M_t-M_{t-1}||_F. No SVD is needed during training; occasional SVDs on sampled layers can diagnose singular-value behavior.

(4) First experiment: train a 100M–300M decoder-only Transformer on a fixed small language-model corpus using the same batch size, learning rate, momentum, and FLOP budget. Compare baseline Muon with fixed t=5, exact-polar Muon if feasible, finite t=2, finite t=3, and the adaptive 2-to-4 schedule. Log training loss versus optimizer FLOPs, gradient/update cosine similarity, orthogonalization residual, and loss spikes after rank or scale changes. Success is lower loss at equal optimizer compute, fewer unstable spikes, or equal perplexity with fewer Newton–Schulz matrix multiplications. A falsification is that shallow finite-depth variants fail to match t=5 perplexity or show no stability improvement over exact polar updates on ill-conditioned or low-rank initialized layers.

## Verification

- Status: mechanism_works
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a finite Newton–Schulz Muon-style matrix optimizer, scalar spectral verification, nuclear/operator duality check, and a fixed-seed ill-conditioned quadratic benchmark. The math checks passed: polar achieves the nuclear-norm optimum exactly, zero remains zero, small singular values are smoothly attenuated, and normalized singular-value error contracts rapidly. In the toy optimization, t=2 achieved final loss 1.78e-6 versus 5.78e-4 for exact polar and used 360 versus 900 Newton–Schulz matrix multiplications relative to t=5; updates were also much smoother than exact polar. This is a promising signal, not evidence of a transformer-scale win: tiny CPU wall time was not faster than exact SVD, and the benchmark is only a synthetic quadratic proxy.

### Mechanism check

- Verdict: Built a finite Newton–Schulz Muon-style matrix optimizer, scalar spectral verification, nuclear/operator duality check, and a fixed-seed ill-conditioned quadratic benchmark. The math checks passed: polar achieves the nuclear-norm optimum exactly, zero remains zero, small singular values are smoothly attenuated, and normalized singular-value error contracts rapidly. In the toy optimization, t=2 achieved final loss 1.78e-6 versus 5.78e-4 for exact polar and used 360 versus 900 Newton–Schulz matrix multiplications relative to t=5; updates were also much smoother than exact polar. This is a promising signal, not evidence of a transformer-scale win: tiny CPU wall time was not faster than exact SVD, and the benchmark is only a synthetic quadratic proxy.
- Confidence: 7/10
- Limitations: No Transformer, language-model corpus, GPU, FLOP-normalized large-model test, rank-changing training, or statistically powered hyperparameter sweep was performed. Exact SVD and Newton–Schulz timings are not representative at this tiny matrix size; the adaptive schedule was only one hand-selected 2-to-4 schedule.

## Artifacts

- [experiment.py](https://synthcore.org/code/10/experiment.py)
- [report.md](https://synthcore.org/code/10/report.md)
- [results.json](https://synthcore.org/code/10/results.json)
- [Download all files as ZIP](https://synthcore.org/download/10)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.