Smooth Spectral Muon

Integrate this into the optimizer update for every trainable two-dimensional weight matrix, especially transformer projection matrices and MLP weights; leave embeddings, biases, and layer-normalization vectors on AdamW or ordinary SGD. Maintain one momentum matrix per selected weight. At each training step, compute the gradient G, update M <- beta*M + G, and replace the exact polar update by a smoothed spectral update. A numerically robust implementation is: (1) compute the thin SVD M=U diag(s) V^T; (2) form s_scaled=s/sqrt(s*s+epsilon); (3) D=U diag(s_scaled) V^T; (4) W <- W - eta*D. For large matrices, benchmark this against Newton-Schulz polar iteration, but use the exact SVD version first as the correctness reference. Epsilon is a scalar or per-layer value; start with epsilon=c times the squared RMS singular value, where RMS_s=(mean_i s_i^2)^(1/2) and c is in {1e-4,1e-3,1e-2}, then optionally EMA-smooth RMS_s across steps. The paper-derived computation is the singular response s_scaled=s/sqrt(s^2+epsilon) and the potential Phi_epsilon; empirical choices are beta, eta, c, and whether epsilon is fixed or adaptive. Log the minimum singular value, maximum update singular value, update Frobenius norm, and Phi_epsilon(M). First test on a 12-layer small Transformer language model such as GPT-2-small or a 100M-parameter decoder on WikiText-103, comparing AdamW, exact Muon, and smoothed Muon at matched optimizer FLOPs. The initial success signal is fewer NaNs and less update-norm variance on deliberately small batches or low-rank gradient settings, followed by lower validation perplexity or faster training-loss reduction at equal steps. Also test epsilon ablations to verify that large epsilon improves stability but excessive epsilon harms convergence by making the update too linear.