{
 "artifacts": [
  {
   "name": "bench_report.json",
   "url": "https://synthcore.org/code/623/bench_report.json"
  },
  {
   "name": "report.md",
   "url": "https://synthcore.org/code/623/report.md"
  },
  {
   "name": "report_bench_2026-08-31T200515.md",
   "url": "https://synthcore.org/code/623/report_bench_2026-08-31T200515.md"
  },
  {
   "name": "results.json",
   "url": "https://synthcore.org/code/623/results.json"
  },
  {
   "name": "smooth_spectral_muon.py",
   "url": "https://synthcore.org/code/623/smooth_spectral_muon.py"
  },
  {
   "name": "stage2_bench.py",
   "url": "https://synthcore.org/code/623/stage2_bench.py"
  }
 ],
 "category": "optimization",
 "description": "Replace the exact matrix-polar normalization in Muon with the smoothed feedback \\(h_\\epsilon(M)=M(M^\\top M+\\epsilon I)^{-1/2}\\). This retains singular-vector-aware updates and approximately unit-normalizes dominant spectral modes, but avoids unstable behavior when the momentum matrix is rank deficient or has tiny singular values.",
 "download_zip": "https://synthcore.org/download/623",
 "formulas_latex": [
  "$$M_k=\\beta M_{k-1}+\\nabla f(W_k),\\qquad W_{k+1}=W_k-\\eta\\,\\operatorname{Polar}(M_k),\\qquad \\operatorname{Polar}(M)=M(M^\\top M)^{-1/2}.$$",
  "$$h_\\epsilon(Z)=Z(Z^\\top Z+\\epsilon I)^{-1/2}.$$",
  "$$Z=U\\operatorname{diag}(\\sigma_i)V^\\top\\quad\\Longrightarrow\\quad h_\\epsilon(Z)=U\\operatorname{diag}\\left(\\frac{\\sigma_i}{\\sqrt{\\sigma_i^2+\\epsilon}}\\right)V^\\top.$$",
  "$$\\Phi_\\epsilon(Z)=\\operatorname{Tr}\\left[(Z^\\top Z+\\epsilon I)^{1/2}-\\sqrt{\\epsilon}I\\right],\\qquad \\nabla\\Phi_\\epsilon(Z)=h_\\epsilon(Z).$$"
 ],
 "id": 1689,
 "implementation": "Integrate this into the optimizer update for every trainable two-dimensional weight matrix, especially transformer projection matrices and MLP weights; leave embeddings, biases, and layer-normalization vectors on AdamW or ordinary SGD. Maintain one momentum matrix per selected weight. At each training step, compute the gradient G, update M \u003c- beta*M + G, and replace the exact polar update by a smoothed spectral update. A numerically robust implementation is: (1) compute the thin SVD M=U diag(s) V^T; (2) form s_scaled=s/sqrt(s*s+epsilon); (3) D=U diag(s_scaled) V^T; (4) W \u003c- W - eta*D. For large matrices, benchmark this against Newton-Schulz polar iteration, but use the exact SVD version first as the correctness reference. Epsilon is a scalar or per-layer value; start with epsilon=c times the squared RMS singular value, where RMS_s=(mean_i s_i^2)^(1/2) and c is in {1e-4,1e-3,1e-2}, then optionally EMA-smooth RMS_s across steps. The paper-derived computation is the singular response s_scaled=s/sqrt(s^2+epsilon) and the potential Phi_epsilon; empirical choices are beta, eta, c, and whether epsilon is fixed or adaptive. Log the minimum singular value, maximum update singular value, update Frobenius norm, and Phi_epsilon(M). First test on a 12-layer small Transformer language model such as GPT-2-small or a 100M-parameter decoder on WikiText-103, comparing AdamW, exact Muon, and smoothed Muon at matched optimizer FLOPs. The initial success signal is fewer NaNs and less update-norm variance on deliberately small batches or low-rank gradient settings, followed by lower validation perplexity or faster training-loss reduction at equal steps. Also test epsilon ablations to verify that large epsilon improves stability but excessive epsilon harms convergence by making the update too linear.",
 "math_summary": "The paper defines the smoothed matrix-polar feedback for a matrix \\(Z\\in\\mathbb{R}^{m\\times n}\\) and \\(\\epsilon\u003e0\\) as \\(h_\\epsilon(Z)=Z(Z^\\top Z+\\epsilon I)^{-1/2}\\). If \\(Z=U\\operatorname{diag}(\\sigma_i)V^\\top\\) is an SVD, then \\(h_\\epsilon(Z)=U\\operatorname{diag}(\\sigma_i/\\sqrt{\\sigma_i^2+\\epsilon})V^\\top\\), so each singular value is transformed by \\(r_\\epsilon(\\sigma)=\\sigma/\\sqrt{\\sigma^2+\\epsilon}\\): it is approximately \\(1\\) for \\(\\sigma\\gg\\sqrt{\\epsilon}\\) and approximately \\(\\sigma/\\sqrt{\\epsilon}\\) near zero. The response is bounded by \\(0\\le r_\\epsilon(\\sigma)\u003c1\\), preventing arbitrarily large normalized components. The direction is the gradient of the spectral energy \\(\\Phi_\\epsilon(Z)=\\operatorname{Tr}[(Z^\\top Z+\\epsilon I)^{1/2}-\\sqrt{\\epsilon}I]\\), namely \\(\\nabla\\Phi_\\epsilon(Z)=h_\\epsilon(Z)\\). In the optimizer, \\(Z=M_k\\) is the momentum matrix, \\(M_k\\) is the exponentially averaged gradient, \\(\\eta\\) is the learning rate, and \\(\\beta\\) is momentum.",
 "math_tags": [
  "spectral-theory",
  "linear-algebra",
  "optimization",
  "dynamical-systems"
 ],
 "ml_areas": [
  "optimizer",
  "transformer",
  "mlp",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.01911",
  "arxiv_url": "https://arxiv.org/abs/2608.01911",
  "summary_what_math_gives_to_ml": "The paper supplies a smooth spectral replacement for the matrix polar factor, preserving the singular-vector geometry of Muon while removing its singularity at rank-deficient momentum matrices. The key transferable asset is the bounded singular-value response \\(\\sigma \\mapsto \\sigma/\\sqrt{\\sigma^2+\\epsilon}\\): large singular modes receive approximately unit polar normalization, while small or noisy modes are attenuated continuously instead of being amplified by an ill-conditioned inverse square root. This suggests a drop-in matrix optimizer for transformer and MLP weight updates, with \\(\\epsilon\\) scheduled or tuned to control noise sensitivity and numerical stability. The spectral potential \\(\\Phi_\\epsilon\\) also gives an explicit energy whose gradient is exactly the update direction, enabling optimizer diagnostics based on spectral dissipation.",
  "title": "A Continuous-Time Analysis of Smoothed Matrix-Polar Spectral Gradient Flows for Muon-Type Optimization",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 4,
  "usefulness": 7
 },
 "solves": [
  "stability",
  "speedup",
  "generalization"
 ],
 "title": "Smooth Spectral Muon",
 "url": "https://synthcore.org/idea/1689/smooth-spectral-muon",
 "verification": {
  "benchmark": {
   "track": "tabular",
   "model": "mlp_tiny",
   "n_seeds": 8,
   "metric_direction": "lower is better",
   "comparison": {
    "baseline_mean": 7.004796802997589,
    "idea_mean": 21.87750768661499,
    "delta_mean": 14.872710883617401,
    "per_seed_diffs": [
     15.58148193359375,
     14.850963592529297,
     16.02706527709961,
     12.929010391235352,
     13.332911254882813,
     15.454879760742188,
     13.974371910095215,
     16.831002712249756
    ],
    "idea_wins": 0,
    "n_pairs": 8,
    "p_value": 0.0081,
    "mde": 1.1379574709779237,
    "mde_rel_pct": 16.24540301427378,
    "verdict": "idea worse (significant)",
    "system_worked": false
   },
   "baseline": {
    "best_cfg": {
     "beta1": 0.85,
     "lr": 0.006,
     "weight_decay": 0
    },
    "sweep": [
     {
      "cfg": {
       "beta1": 0.85,
       "lr": 0.0015,
       "weight_decay": 0
      },
      "mean": 12.617201328277588
     },
     {
      "cfg": {
       "beta1": 0.95,
       "lr": 0.0015,
       "weight_decay": 0
      },
      "mean": 15.512621641159058
     },
     {
      "cfg": {
       "beta1": 0.85,
       "lr": 0.003,
       "weight_decay": 0
      },
      "mean": 8.035151362419128
     },
     {
      "cfg": {
       "beta1": 0.95,
       "lr": 0.003,
       "weight_decay": 0
      },
      "mean": 12.083711862564087
     },
     {
      "cfg": {
       "beta1": 0.85,
       "lr": 0.006,
       "weight_decay": 0
      },
      "mean": 6.865306854248047
     },
     {
      "cfg": {
       "beta1": 0.95,
       "lr": 0.006,
       "weight_decay": 0
      },
      "mean": 8.458353757858276
     }
    ],
    "full": {
     "mean": 7.004796802997589,
     "std": 0.3739293172768815,
     "per_seed": [
      6.71429443359375,
      7.094020843505859,
      6.866542816162109,
      6.786369323730469,
      6.342711448669434,
      7.308849334716797,
      7.395407676696777,
      7.530178546905518
     ],
     "n": 8
    }
   },
   "idea": {
    "mean": 21.87750768661499,
    "std": 1.4977844581209827,
    "per_seed": [
     22.2957763671875,
     21.944984436035156,
     22.89360809326172,
     19.71537971496582,
     19.675622940063477,
     22.763729095458984,
     21.369779586791992,
     24.361181259155273
    ],
    "n": 8
   },
   "mechanism_signature": {
    "confirmed": true,
    "idea_grid": [
     {
      "cfg": {
       "beta": 0.9,
       "c": 0.001,
       "lr": 0.0015
      },
      "result": {
       "mean": 105.54728984832764,
       "n": 8,
       "per_seed": [
        114.30227661132812,
        109.72885131835938,
        105.97409057617188,
        96.68315124511719,
        103.86304473876953,
        104.20171356201172,
        99.61021423339844,
        110.01497650146484
       ],
       "std": 5.39382412585861
      }
     },
     {
      "cfg": {
       "beta": 0.9,
       "c": 0.001,
       "lr": 0.003
      },
      "result": {
       "mean": 43.18799352645874,
       "n": 8,
       "per_seed": [
        49.57056427001953,
        46.739295959472656,
        42.695228576660156,
        37.679901123046875,
        39.12871551513672,
        43.38615036010742,
        39.62131118774414,
        46.68278121948242
       ],
       "std": 3.9588922932446455
      }
     },
     {
      "cfg": {
       "beta": 0.9,
       "c": 0.001,
       "lr": 0.006
      },
      "result": {
       "mean": 21.87750768661499,
       "n": 8,
       "per_seed": [
        22.2957763671875,
        21.944984436035156,
        22.89360809326172,
        19.71537971496582,
        19.675622940063477,
        22.763729095458984,
        21.369779586791992,
        24.361181259155273
       ],
       "std": 1.4977844581209827
      }
     }
    ],
    "mechanism_signature": {
     "confirmed": true,
     "n_observations": 600,
     "observed_response_mean": 0.3883247250989132,
     "predicted_response_mean": 0.3883231416550379,
     "prediction": "observed singular response follows t/sqrt(t^2+1), t=s/sqrt(epsilon)",
     "response_mae": 0.0000017781029070577825,
     "update_norm_cv": 0.08146856078007694
    },
    "track_rationale": "Optimizer modification belongs on Friedman#1 tabular MLP; architecture and data are shared."
   },
   "custom_track": null
  },
  "mechanism_confirmed": true,
  "peer_reviewed": false,
  "practical_verdict": "harms",
  "stage1_mechanism_check": {
   "worked": true,
   "confidence": 9,
   "verdict": "Built and numerically verified Smooth Spectral Muon with a thin-SVD implementation, spectral potential, finite-difference gradient check, and toy low-rank matrix regression. The predicted transition and asymptotic scaling matched essentially exactly: transition at sigma/sqrt(epsilon)=1, small-mode ratio 0.999999995, large-mode asymptotic 1.000000016, bounded response below 1, and matrix spectral error 0. The smoothed optimizer retained Muon-like loss while providing finite update variability, but did not beat SGD, so this confirms the mechanism rather than an overall training win.",
   "metrics": {
    "baseline": "Toy SGD final loss 2.8356e-05, best loss 3.7964e-06, update-norm CV 0.4468; exact Muon final loss 1.8187e-04, best loss 1.2352e-04, update-norm CV 3.0e-16.",
    "idea": "Smooth Muon with c=1e-3 final loss 1.6260e-04, best loss 1.2357e-04, update-norm CV 0.0193; finite-difference gradient error 2.95e-11; predicted response transition observed exactly at ratio 1.0 with response 0.70710678."
   },
   "how_to_run": "python3 smooth_spectral_muon.py",
   "files": [
    "smooth_spectral_muon.py",
    "results.json",
    "run_output.txt"
   ],
   "limitations": "Only a small NumPy low-rank matrix-regression toy was tested. No Transformer, language modeling, WikiText, AdamW, CUDA benchmark, matched-FLOP study, NaN stress test, or multi-seed statistical evaluation was performed."
  },
  "status": "mech_ok_no_baseline",
  "status_label": "Mechanism confirmed, baseline not beaten",
  "updated_at": "2026-09-02T12:30:10",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": true,
    "tested": true
   },
   "practical_benchmark": {
    "beats_baseline": false,
    "tested": true,
    "verdict": "harms"
   },
   "toy_mechanism_gate": {
    "confirmed": true,
    "tested": true
   }
  }
 }
}