# Polar-Muon Stiefel Optimizer

- ID: 1882
- Canonical URL: https://synthcore.org/idea/1882/polar-muon-stiefel-optimizer
- API JSON: https://synthcore.org/api/idea/1882.json
- API Markdown: https://synthcore.org/api/idea/1882.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.06218](https://arxiv.org/abs/2608.06218)
- Category: optimization
- Solves: stability, accuracy, speedup
- ML areas: optimizer, architecture, training
- Math tags: optimization, linear-algebra, geometry
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Use the negative polar factor of a tangent-projected momentum matrix as the update direction for a weight matrix constrained to the Stiefel manifold. After taking the step, apply a polar retraction so the columns remain exactly orthonormal, avoiding penalty losses and constraint drift.

## Mathematical statement

The paper's linear minimization oracle solves $\min_{\|B\|_2\leq 1}\langle M,B\rangle$, where $M\in\mathbb{R}^{m\times n}$ is the momentum matrix, $B$ is the update matrix, and $\|\cdot\|_2$ is the spectral norm. If $M=U\Sigma V^\top$ is a thin singular-value decomposition, the exact optimizer is $B^\star=-UV^\top=-\operatorname{polar}(M)$. For a Stiefel parameter $X\in\mathbb{R}^{m\times n}$ satisfying $X^\top X=I_n$, define $\operatorname{sym}(A)=(A+A^\top)/2$ and the tangent projection $P_X(M)=M-X\operatorname{sym}(X^\top M)$. This satisfies $X^\top P_X(M)+P_X(M)^\top X=0$. The adapted direction is $D=-\operatorname{polar}(P_X(M))$. Feasibility is restored using the polar retraction $R_X(\eta D)=\operatorname{polar}(X+\eta D)=(X+\eta D)((X+\eta D)^\top(X+\eta D))^{-1/2}$, assuming full column rank. In practice, if $Z=U\Sigma V^\top$ is a thin SVD, compute $\operatorname{polar}(Z)=UV^\top$.

## Key formulas

- $$\min_{\|B\|_2\leq 1}\langle M,B\rangle,\qquad M=U\Sigma V^\top\quad\Longrightarrow\quad B^\star=-UV^\top=-\operatorname{polar}(M).$$
- $$P_X(M)=M-X\operatorname{sym}(X^\top M),\qquad \operatorname{sym}(A)=\frac{A+A^\top}{2},\qquad X^\top P_X(M)+P_X(M)^\top X=0.$$
- $$D_t=-\operatorname{polar}(P_{X_t}(M_t)),\qquad X_{t+1}=\operatorname{polar}(X_t+\eta_tD_t)=U_{t+1}V_{t+1}^\top.$$
- $$M_t=\beta M_{t-1}+(1-\beta)\nabla_X\mathcal{L}(X_t),\qquad \operatorname{polar}(Z)=UV^\top\ \text{for the thin SVD }Z=U\Sigma V^\top.$$

## Implementation notes

(1) Integration point: replace the optimizer update for each rectangular neural-network weight matrix $X\in\mathbb{R}^{m\times n}$ whose columns are intended to remain orthonormal. Suitable targets include tall linear layers, orthogonal projection layers, recurrent transition matrices, and low-rank adapter factors. Do not apply the update to biases or unconstrained parameters. Store a full-precision momentum matrix $M$ with the same shape as $X$.

(2) Pseudocode:
```text
initialize X with orthonormal columns and M = 0
for each minibatch:
    loss = model(batch, X)
    G = autograd(loss, X)
    M = beta*M + (1-beta)*G
    S = 0.5*(X.T @ M + M.T @ X)
    T = M - X @ S
    U, Sigma, Vt = thin_svd(T)
    D = -(U @ Vt)
    Z = X + eta*D
    U2, Sigma2, Vt2 = thin_svd(Z)
    X = U2 @ Vt2
```
The line defining $T$ is the tangent projection $P_X(M)$; the line defining $D$ is the paper's spectral-norm linear-minimization solution; the final SVD is the polar retraction. Use bias correction if desired, but keep the first comparison simple.

(3) Computed directly from the mathematics: the tangent projection, polar direction, and exact orthonormal retraction. Estimated empirically: the learning rate $\eta$, momentum coefficient $\beta$, whether tangent projection is necessary, and the tradeoff between SVD cost and optimization quality. Log $\|X^\top X-I\|_F$, the singular values of $X$, training loss, validation loss, gradient norms, wall-clock time, and SVD FLOPs.

(4) First cheap experiment: train a four-layer width-256 MLP on MNIST or CIFAR-10, making two dense layers Stiefel-constrained. Compare AdamW with an orthogonality penalty, ordinary Muon followed by QR, and this optimizer at matched batch size and approximately matched compute. Sweep $\eta\in\{10^{-3},3\times10^{-3},10^{-2}\}$ and $\beta\in\{0.9,0.95,0.99\}$. A positive result is lower validation loss at equal optimizer steps, near-machine-precision orthogonality without penalty tuning, and equal or better final accuracy. Also report wall-clock performance; repeated SVDs may erase any optimization gain.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built the Polar-Muon Stiefel update with tangent projection, negative polar oracle direction, and polar retraction. The three mechanism predictions were confirmed: tangent residual was 2.56e-15, oracle objective matched -||M||_* with absolute error at most 1.4e-14 and linear scaling, and retraction maintained unit singular values with orthogonality error 2.8e-15–3.6e-15 for eta from 1e-4 to 10. In the toy optimization, Polar-Muon reached loss -37.0577 with machine-precision feasibility, comparable to normalized Riemannian GD (-37.0710) but slower; the penalty baseline reached lower unconstrained loss while drifting substantially.

### Mechanism check

- Verdict: Built the Polar-Muon Stiefel update with tangent projection, negative polar oracle direction, and polar retraction. The three mechanism predictions were confirmed: tangent residual was 2.56e-15, oracle objective matched -||M||_* with absolute error at most 1.4e-14 and linear scaling, and retraction maintained unit singular values with orthogonality error 2.8e-15–3.6e-15 for eta from 1e-4 to 10. In the toy optimization, Polar-Muon reached loss -37.0577 with machine-precision feasibility, comparable to normalized Riemannian GD (-37.0710) but slower; the penalty baseline reached lower unconstrained loss while drifting substantially.
- Confidence: 9/10
- Limitations: Only a small NumPy linear-objective Stiefel toy problem was tested. No neural-network, MNIST/CIFAR, AdamW, full Muon, GPU, hyperparameter sweep, matched-FLOP study, or repeated-seed statistical comparison was performed; random feasible competitors provide a sanity check rather than a formal optimality proof.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_polar_muon.py](https://synthcore.org/code/718/bench_polar_muon.py)
- [bench_report.json](https://synthcore.org/code/718/bench_report.json)
- [experiment.py](https://synthcore.org/code/718/experiment.py)
- [report.md](https://synthcore.org/code/718/report.md)
- [report_bench_2026-09-01T011357.md](https://synthcore.org/code/718/report_bench_2026-09-01T011357.md)
- [results.json](https://synthcore.org/code/718/results.json)
- [Download all files as ZIP](https://synthcore.org/download/718)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.