✗ Failed on benchmark 2026

Descent-Certified LMO Sign Switching

Implementation & benchmark of arXiv:2607.29674 — Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback

Usefulness7/10

Difficulty4/10

Novelty7/10

Stability Stability Accuracy Scalability Optimizer Training Quantization Optimization Convex analysis Linear algebra Dynamical systems

Source paper: Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback arXiv:2607.29674 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Keep the empirically effective post-LMO sign update, but reject it whenever a fresh minibatch estimates that it is poorly aligned with the gradient. Fall back to the gradient-side error-feedback candidate in those cases. This converts the paper's constructive divergence warning into an inexpensive runtime safeguard rather than assuming that any sign placement is universally safe.

Formulas

$$D_t^{post}=\operatorname{sign}\!\left(\mathcal{L}(M_t)\right),\qquad D_t^{pre}=\mathcal{L}\!\left(\operatorname{sign}(M_t+R_t)\right),$$

$$F(W-\eta D)\le F(W)-\eta\langle\nabla F(W),D\rangle_F+\frac{L\eta^2}{2}\|D\|_F^2,$$

$$\rho(D)=\frac{\langle\widehat G,D\rangle_F}{\|\widehat G\|_F\,\|D\|_F+\epsilon},\qquad D_t=\begin{cases}D_t^{post},&\rho(D_t^{post})\ge\tau,\\D_t^{pre},&\rho(D_t^{post})<\tau.\end{cases}$$

Mathematical statement

Let M_t be the Muon momentum matrix. Define the post-LMO candidate D_t^{post}=sign(L(M_t)) and the pre-LMO candidate D_t^{pre}=L(sign(M_t+R_t)), where R_t is a gradient-side error-feedback residual and L is the polar LMO. For an L-smooth objective F, the descent lemma states F(W-eta D) <= F(W)-eta<grad F(W),D>_F+(L eta^2/2)||D||_F^2. Here L is a local or global smoothness constant, eta is the stepsize, and ||.||_F is the Frobenius norm. Since L is generally unavailable, estimate the gradient with an independent minibatch Ghat and use the scale-free alignment rho(D)=<Ghat,D>_F/(||Ghat||_F||D||_F+epsilon). Select the post-LMO candidate only when rho(D_post) exceeds a threshold tau; otherwise use the pre-LMO candidate. Positive alignment is not a formal global guarantee under stochastic gradients, but it is a directly testable approximation to the first-order descent condition and targets exactly the uphill instances identified by the paper.

Implementation notes

Integrate the switch immediately before the parameter update for each 2-D matrix in Muon. Compute the usual momentum matrix M and retain a gradient-side residual R. Generate two candidates without changing the baseline learning-rate scale: D_post = sign(polar(M)) and D_pre = polar(sign(M + R)). Obtain an independent gradient estimate Ghat from a small held-out minibatch, a second accumulation microbatch, or a one-step delayed gradient; using an independent estimate is preferable because reusing the exact minibatch can overfit the routing decision. Compute rho = frobenius_inner(Ghat, D_post) / (frobenius_norm(Ghat) * frobenius_norm(D_post) + 1e-8). If rho >= tau, apply D_post; otherwise apply D_pre and update the pre-LMO residual with R = (M + R) - sign(M + R). Sweep tau over 0, 0.02, 0.05, and 0.1. The descent lemma supplies the first-order/second-order rationale; estimate gradient alignment empirically and do not assume that the unknown smoothness constant L is known. Log the branch-selection fraction, alignment distributions, one-step loss changes, and update norms. First test CIFAR-10 with ResNet-18 and a 6-layer nanoGPT, comparing full Muon, post-LMO SignMuon, pre-LMO error-feedback SignMuon, and the switch at matched compute and communication. Success is preservation of most post-LMO sign's loss descent together with fewer positive one-step loss changes, reduced loss spikes, and improved worst-seed validation accuracy.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built exact SVD-polar post/pre candidates with independent noisy alignment gating and residual feedback. Three quantitative checks manifested: observed routing fractions exactly matched P(rho_hat >= tau), the linear loss change matched -eta< G,D > to numerical precision across all tested stepsizes, and routing agreement decreased monotonically from 1.00 to 0.765 as held-out gradient noise increased. In the matched noisy quadratic run, switching reduced uphill steps from 59 to 57 and improved best loss from 0.0421 to 0.0404 versus post-LMO, but its final loss 0.183 was worse than post-LMO’s 0.144; therefore this supports the safeguard mechanism, not a decisive overall optimization win.

Agent confidence: 8/10

Baseline: Post-LMO SignMuon: final_loss=0.14447, best_loss=0.04207, uphill_steps=59/160; post alignment negative rate=0.346.
Idea: Descent-certified switch at tau=0.05: final_loss=0.18320, best_loss=0.04038, uphill_steps=57/160, post-branch fraction=0.750; selected-direction negative true-alignment rate=0.0206 versus 0.346 for post-only. Threshold routing fractions at tau=[0,0.02,0.05,0.1] were [0.659,0.656,0.653,0.647], exactly matching empirical predicted fractions. Linear descent identity max absolute error across eta sweep=1.11e-16; routing agreement at held-out noise [0,0.25,0.55,1,2]=[1.000,0.974,0.937,0.882,0.765].

Limitations: Only a 4x4 synthetic matrix and one matched-seed noisy quadratic comparison were tested; no CIFAR-10, nanoGPT, ResNet-18, multi-seed statistics, wall-clock/communication measurement, or GPU implementation was evaluated. The held-out gradient was simulated by additive noise rather than produced by an actual independent minibatch, and the threshold sweep’s branch-fraction prediction is definitional rather than an independent statistical model.

How to run: python3 switch_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.29674) Aug 31, 2026

No significant difference vs. the tuned baseline (+0.1%, p=0.8426, wins 3 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 21.575
Idea mean: 0
Effect (Δ): 0.0109 (+0.1%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.8426 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 0 configs

Limitations:

Only the structurally appropriate built-in tabular track was tested, not CIFAR, sequence, or dynamics. Runtime and communication overhead were not separately benchmarked; the implementation uses a compact custom SignMuon-style loop rather than the bench Adam train_model path because the idea changes the optimizer itself.

How to run: python3 bench_switch.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented Descent-Certified LMO Sign Switching as a local optimizer training loop for the structurally appropriate tabular optimizer track, with post-LMO baseline, pre-LMO residual feedback, independent held-out minibatch alignment, and matched hyperparameter sweeps. Across 8 paired seeds, the best idea configuration had MSE 21.5859 versus baseline 21.5750, paired delta +0.0109 and permutation p=0.8426, so there was no significant improvement. The trained-model mechanism signature was confirmed, but it did not translate into a task-metric win.

Baseline: tabular/mlp_tiny post-LMO SignMuon, best lr=0.006 beta=0.9: mean test MSE=21.575043, std=1.915759, per-seed=[22.023918,17.909542,21.564068,20.560976,20.688057,23.367746,21.641396,24.844645]
Idea: Descent-certified switch, best lr=0.006 beta=0.9 tau=0.0: mean test MSE=21.585942, std=1.884198, per-seed=[22.238335,17.963984,21.634672,20.441940,20.853069,23.095635,21.598305,24.861594]; delta=+0.010898, p=0.8426, wins=3/8; mechanism confirmed with post_fraction=0.960648, predicted_fraction=0.960648, post_negative=0.039352, chosen_negative=0.033069

Benchmark result