△ Mechanism confirmed, baseline not beaten 2026

Support-Identified Newton Optimizer for Sparse Orthogonal Layers

Implementation & benchmark of arXiv:2607.12877 — From Manifold Identification to Newton Acceleration on Intersections: Sparse Stiefel Optimization

Usefulness7/10

Difficulty6/10

Novelty7/10

Optimization Speedup Stability Memory Optimizer Attention Ssm Pruning Optimization Differential geometry Linear algebra Dynamical systems

Source paper: From Manifold Identification to Newton Acceleration on Intersections: Sparse Stiefel Optimization arXiv:2607.12877 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Train a matrix-valued neural layer under an exact or near-exact Stiefel constraint while using an l1 or row-group sparsity penalty. During early training, use manifold proximal-gradient steps to identify a stable nonzero support; once the support stops changing, switch to Newton-CG steps restricted to the smooth intersection of the Stiefel tangent space and the fixed-support subspace. This can reduce the number of optimizer iterations needed to obtain sparse, well-conditioned projections.

Formulas

$$\min_X F(X):=f(X)+g(X),\qquad \mathrm{s.t.}\qquad X\in\mathrm{St}(n,r),$$

$$\mathrm{St}(n,r):=\{X\in\mathbb{R}^{n\times r}\mid X^\top X=I_r\},\qquad g(X)=\lambda\|X\|_1=\lambda\sum_{i=1}^n\sum_{j=1}^r|X_{ij}|,$$

$$T_X\mathrm{St}(n,r)=\{Z\in\mathbb{R}^{n\times r}:X^\top Z+Z^\top X=0\},\qquad T_X\mathcal{M}_S=T_X\mathrm{St}(n,r)\cap E_S,$$

$$\texttt{accepted/superlinear}:\ t_{k+1}=t_k,\ \lambda_{k+1}=\lambda_k^+;\qquad \texttt{accepted/linear/full}:\ t_{k+1}=t_k^+,\ \lambda_{k+1}=\lambda_k^-;\qquad \texttt{accepted/backtracked}:\ t_{k+1}=t_k^-,\ \lambda_{k+1}=\lambda_k^+.$$

Mathematical statement

The paper studies min_X F(X)=f(X)+g(X) subject to X in St(n,r), where St(n,r)={X in R^{n times r}: X^T X=I_r}, f is smooth with Lipschitz Euclidean gradient, and g is convex and Lipschitz, including g(X)=lambda||X||_1 and g(X)=lambda||X||_{2,1}. Let S be the currently identified support and E_S={Z: Z_{ij}=0 whenever (i,j) is not in S} for entry sparsity, or E_S={Z: Z_{i,:}=0 whenever row i is not in S} for row sparsity. The Stiefel tangent space is T_X St(n,r)={Z: X^T Z+Z^T X=0}, so the identified tangent model is T_X St(n,r) intersect E_S. After support identification, Newton-CG approximately solves H_k[Z]=-grad_{M_S} f(X_k) on this restricted tangent model, where H_k is the Hessian of the smooth restriction and grad_{M_S} is the projected gradient. The safeguard uses proximal manifold-gradient steps when projection or Newton fails. The extracted acceptance logic is: an accepted superlinear Newton step keeps t_k unchanged and decreases lambda_k; an accepted linear full step increases t_k and decreases lambda_k; an accepted backtracked step decreases t_k and increases lambda_k. Under regularity and a second-order sufficient condition, the active manifold is identified finitely and the local Newton sequence has Q-superlinear convergence.

Implementation notes

(1) Integration point: replace the standard optimizer for a matrix W in one orthogonal linear layer, attention projection, or SSM transition matrix. Parameterize the forward weight directly as X and initialize it with X^T X=I_r; use entrywise sparsity or row-group sparsity. Let f(X) be the minibatch loss and g(X)=lambda||X||_1 or lambda||X||_{2,1}. (2) Pseudocode: at iteration k, compute a manifold proximal-gradient candidate Y=Retr_{X_k}(-alpha_k grad f(X_k)), apply soft thresholding, and retract back toward Y^T Y=I. Define S_k={(i,j): |X_{k,ij}|>tau}, or active rows for group sparsity. If S_k is unchanged for q consecutive iterations, form the tangent operator on T_{X_k}St intersect E_{S_k} and solve H_k[Z]=-grad_{M_{S_k}}f(X_k) with truncated CG. Retract X_{k+1}=Retr_{X_k}(eta Z) and test Armijo decrease. If CG detects negative curvature, the residual is too large, or retraction fails, use the proximal-gradient candidate. Increase the trust scale after accepted full linear steps, preserve it after superlinear steps, damp after backtracking, decrease sparsity regularization after successful full Newton steps, and increase it after backtracking. (3) Compute from the mathematics: tangent constraints, support-restricted CG, Stiefel retraction, and acceptance tests. Estimate empirically: support stabilization, local convergence slope, and Hessian-vector products through autodiff. (4) First experiment: train a 2-layer MLP or small Transformer on CIFAR-10 or WikiText-2 with a 256x256 sparse orthogonal projection. Compare AdamW plus soft thresholding, projected Riemannian gradient, and this ManPG/Newton-CG optimizer at matched forward/backward FLOPs. Measure wall-clock time to a fixed validation loss, nonzero count, ||X^T X-I||_F, loss decrease per optimizer step, and whether the final loss curve becomes superlinear-like. A useful success signal is 20–30% fewer optimizer steps at the same sparsity and validation accuracy without orthogonality drift.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and ran a self-contained support-restricted Riemannian Newton experiment for the r=1 Stiefel case (the unit sphere), including retraction, tangent-space restriction, Newton solves, safeguards, and support stabilization. The mechanism manifested: soft-threshold support counts exactly matched the predicted transition rule across all six lambda values, tangent dimensions matched |S|-1 with residuals below 1.8e-16, and Newton errors showed quadratic ratios near 0.51 while gradient errors had linear ratios near 0.965. In the toy comparison, support-identified Newton reached the final-objective target in 45 steps versus 60 for projected gradient, with both maintaining orthogonality below 5e-16; this is a promising toy signal, not evidence of a general neural-network speedup.

Agent confidence: 8/10

Baseline: Projected Riemannian gradient: 60 steps to final objective + 1e-6; final objective -0.0800197749; orthogonality error 4.44e-16; final support size 7.
Idea: Support-identified Newton: 45 steps to final objective + 1e-6, 25% fewer; final objective -0.0800197764; orthogonality error 1.11e-16; support switch detected; final support size 7. Support sweep observed counts [6,5,5,4,3,2] exactly matched predictions. Newton first-step error-squared ratios across initial errors 0.1, 0.05, 0.02, 0.01 were [0.523,0.517,0.512,0.510], versus gradient linear ratios [0.969,0.966,0.965,0.964].

Limitations: Only a synthetic r=1 sphere problem was tested, not a matrix St(n,r) layer with r>1, row-group sparsity, minibatches, autodiff Hessian-vector products, GPU execution, or CIFAR/Transformer training. The sparsity comparison uses a fixed active set rather than a full proximal-manifold implementation with dynamically changing l1 support, and wall-clock speed was not evaluated.

How to run: python3 support_newton_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.12877) Sep 2, 2026

Significantly WORSE than the tuned baseline (+2711.5%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 8.3277
Idea mean: 234.138
Effect (Δ): 225.8103 (+2711.5%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Only the registered tabular track and mlp_tiny were tested; vision, sequence, and dynamics transfer were not evaluated. The implementation used row-group support and a small truncated Newton-CG approximation rather than a full production fixed-entry-support solver, and FLOP-normalized speed was not established.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

On the registered tabular Friedman#1/mlp_tiny benchmark, support-identified restricted Newton preserved the Stiefel constraint and stabilized support, but it substantially degraded task performance. Baseline test MSE was 8.3277 versus 234.1380 for the idea, with paired delta +225.8103 and permutation p=0.0081; this is a significant loss, not a win.

Baseline: best_cfg lr=0.01, lam=0.0; mean test MSE 8.327727615833282; std 1.107429741064248.
Idea: best_cfg lr=0.001, lam=0.0; mean test MSE 234.13803672790527; std 8.541568203357285; per-seed [245.36729431152344,229.77072143554688,242.98947143554688,216.77557373046875,240.6185302734375,231.03858947753906,231.6000518798828,234.94406127929688].
Mechanism signature: map[confirmed:true observed_mean_newton_steps:148 observed_mean_orthogonality:5.595025101001738e-07 observed_mean_support_changes:0 observed_newton_switch_rate:1]

Benchmark result