# Descent-Certified LMO Sign Switching

- ID: 1616
- Canonical URL: https://synthcore.org/idea/1616/descent-certified-lmo-sign-switching
- API JSON: https://synthcore.org/api/idea/1616.json
- API Markdown: https://synthcore.org/api/idea/1616.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.29674](https://arxiv.org/abs/2607.29674)
- Category: stability
- Solves: stability, accuracy, scalability
- ML areas: optimizer, training, quantization
- Math tags: optimization, convex-analysis, linear-algebra, dynamical-systems
- Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10

## Idea description

Keep the empirically effective post-LMO sign update, but reject it whenever a fresh minibatch estimates that it is poorly aligned with the gradient. Fall back to the gradient-side error-feedback candidate in those cases. This converts the paper's constructive divergence warning into an inexpensive runtime safeguard rather than assuming that any sign placement is universally safe.

## Mathematical statement

Let M_t be the Muon momentum matrix. Define the post-LMO candidate D_t^{post}=sign(L(M_t)) and the pre-LMO candidate D_t^{pre}=L(sign(M_t+R_t)), where R_t is a gradient-side error-feedback residual and L is the polar LMO. For an L-smooth objective F, the descent lemma states F(W-eta D) <= F(W)-eta<grad F(W),D>_F+(L eta^2/2)||D||_F^2. Here L is a local or global smoothness constant, eta is the stepsize, and ||.||_F is the Frobenius norm. Since L is generally unavailable, estimate the gradient with an independent minibatch Ghat and use the scale-free alignment rho(D)=<Ghat,D>_F/(||Ghat||_F||D||_F+epsilon). Select the post-LMO candidate only when rho(D_post) exceeds a threshold tau; otherwise use the pre-LMO candidate. Positive alignment is not a formal global guarantee under stochastic gradients, but it is a directly testable approximation to the first-order descent condition and targets exactly the uphill instances identified by the paper.

## Key formulas

- $$D_t^{post}=\operatorname{sign}\!\left(\mathcal{L}(M_t)\right),\qquad D_t^{pre}=\mathcal{L}\!\left(\operatorname{sign}(M_t+R_t)\right),$$
- $$F(W-\eta D)\le F(W)-\eta\langle\nabla F(W),D\rangle_F+\frac{L\eta^2}{2}\|D\|_F^2,$$
- $$\rho(D)=\frac{\langle\widehat G,D\rangle_F}{\|\widehat G\|_F\,\|D\|_F+\epsilon},\qquad D_t=\begin{cases}D_t^{post},&\rho(D_t^{post})\ge\tau,\\D_t^{pre},&\rho(D_t^{post})<\tau.\end{cases}$$

## Implementation notes

Integrate the switch immediately before the parameter update for each 2-D matrix in Muon. Compute the usual momentum matrix M and retain a gradient-side residual R. Generate two candidates without changing the baseline learning-rate scale: `D_post = sign(polar(M))` and `D_pre = polar(sign(M + R))`. Obtain an independent gradient estimate Ghat from a small held-out minibatch, a second accumulation microbatch, or a one-step delayed gradient; using an independent estimate is preferable because reusing the exact minibatch can overfit the routing decision. Compute `rho = frobenius_inner(Ghat, D_post) / (frobenius_norm(Ghat) * frobenius_norm(D_post) + 1e-8)`. If `rho >= tau`, apply D_post; otherwise apply D_pre and update the pre-LMO residual with `R = (M + R) - sign(M + R)`. Sweep tau over 0, 0.02, 0.05, and 0.1. The descent lemma supplies the first-order/second-order rationale; estimate gradient alignment empirically and do not assume that the unknown smoothness constant L is known. Log the branch-selection fraction, alignment distributions, one-step loss changes, and update norms. First test CIFAR-10 with ResNet-18 and a 6-layer nanoGPT, comparing full Muon, post-LMO SignMuon, pre-LMO error-feedback SignMuon, and the switch at matched compute and communication. Success is preservation of most post-LMO sign's loss descent together with fewer positive one-step loss changes, reduced loss spikes, and improved worst-seed validation accuracy.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built exact SVD-polar post/pre candidates with independent noisy alignment gating and residual feedback. Three quantitative checks manifested: observed routing fractions exactly matched P(rho_hat >= tau), the linear loss change matched -eta< G,D > to numerical precision across all tested stepsizes, and routing agreement decreased monotonically from 1.00 to 0.765 as held-out gradient noise increased. In the matched noisy quadratic run, switching reduced uphill steps from 59 to 57 and improved best loss from 0.0421 to 0.0404 versus post-LMO, but its final loss 0.183 was worse than post-LMO’s 0.144; therefore this supports the safeguard mechanism, not a decisive overall optimization win.

### Mechanism check

- Verdict: Built exact SVD-polar post/pre candidates with independent noisy alignment gating and residual feedback. Three quantitative checks manifested: observed routing fractions exactly matched P(rho_hat >= tau), the linear loss change matched -eta< G,D > to numerical precision across all tested stepsizes, and routing agreement decreased monotonically from 1.00 to 0.765 as held-out gradient noise increased. In the matched noisy quadratic run, switching reduced uphill steps from 59 to 57 and improved best loss from 0.0421 to 0.0404 versus post-LMO, but its final loss 0.183 was worse than post-LMO’s 0.144; therefore this supports the safeguard mechanism, not a decisive overall optimization win.
- Confidence: 8/10
- Limitations: Only a 4x4 synthetic matrix and one matched-seed noisy quadratic comparison were tested; no CIFAR-10, nanoGPT, ResNet-18, multi-seed statistics, wall-clock/communication measurement, or GPU implementation was evaluated. The held-out gradient was simulated by additive noise rather than produced by an actual independent minibatch, and the threshold sweep’s branch-fraction prediction is definitional rather than an independent statistical model.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.8426
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/603/bench_report.json)
- [bench_switch.py](https://synthcore.org/code/603/bench_switch.py)
- [report.md](https://synthcore.org/code/603/report.md)
- [report_bench_2026-08-31T194227.md](https://synthcore.org/code/603/report_bench_2026-08-31T194227.md)
- [results.json](https://synthcore.org/code/603/results.json)
- [switch_experiment.py](https://synthcore.org/code/603/switch_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/603)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.