✗ Failed on benchmark 2026

Low-Rank Curvature-Scaled Saddle Optimizer

Implementation & benchmark of arXiv:2607.03030 — Subspace curvature-scaling high-index saddle dynamics for accelerating ill-conditioned saddle point searches

Usefulness7/10

Difficulty6/10

Novelty7/10

Optimization Speedup Stability Optimizer Training dynamics Optimization Dynamical systems Linear algebra Spectral theory

Source paper: Subspace curvature-scaling high-index saddle dynamics for accelerating ill-conditioned saddle point searches arXiv:2607.03030 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the sign-flip-only dynamics of high-index saddle search with low-rank inverse-curvature scaling on the estimated negative-curvature subspace. Directions with small negative Hessian eigenvalues then receive approximately curvature-independent updates instead of extremely slow updates proportional to their tiny curvature.

Formulas

$$\left\{\begin{aligned} \frac{\mathrm{d}x}{\mathrm{d}t}&=-\beta\left(I-\sum_{i=1}^{k}2v_{i}v_{i}^{\top}\right)\nabla E(x),\\ \frac{\mathrm{d}v_{i}}{\mathrm{d}t}&=-\gamma\left(I-v_{i}v_{i}^{\top}-\sum_{j=1}^{i-1}2v_{j}v_{j}^{\top}\right)\nabla^{2}E(x)v_{i},\qquad 1\leqslant i\leqslant k.\end{aligned}\right.$$

$$\lambda_i=v_i^\top H v_i,\qquad H=\nabla^2E(x),\qquad V=[v_1,\ldots,v_k],\qquad V^\top V=I_k.$$

$$\dot{x}_{\mathrm{SCS}}=-\beta(I-VV^\top)g+\beta\sum_{i=1}^{k}\frac{1}{|\lambda_i|+\epsilon}v_i v_i^\top g,\qquad g=\nabla E(x).$$

$$Hv_i=\lambda_i v_i,\ \lambda_i<0\quad\Longrightarrow\quad \dot{a}_i=-\beta\frac{|\lambda_i|}{|\lambda_i|+\epsilon}a_i,\qquad x-x^*=a_i v_i.$$

Mathematical statement

Equation (1) defines high-index saddle dynamics for an energy E(x): x is the parameter vector, g=
abla E(x), H=
abla^2E(x), v_1,\ldots,v_k are orthonormal approximations to the k unstable Hessian eigenvectors, and \beta,\gamma>0 are relaxation parameters. The position update is \dot{x}=-\beta(I-2\sum_{i=1}^k v_i v_i^ op)g, which reflects the gradient on selected eigendirections. Let \lambda_i=v_i^ op H v_i<0 be the Rayleigh curvature of v_i. The transferred method applies a damped inverse curvature to each selected direction: the orthogonal complement uses ordinary descent, while direction v_i uses multiplier (|\lambda_i|+\epsilon)^{-1}. Here V=[v_1,\ldots,v_k] has orthonormal columns and \epsilon>0 is a damping floor. For an exact eigenvector, the linearized coordinate rate becomes -\beta|\lambda_i|/(|\lambda_i|+\epsilon), so the rate no longer vanishes as |\lambda_i| becomes small compared with the other negative curvatures.

Implementation notes

Integrate this at the parameter-update stage of a saddle-search optimizer, initially on a small MLP or transformer block with scalar objective E( heta). At iteration t, compute g=
abla_ heta E( heta_t). Use k-step Lanczos or LOBPCG with Hessian-vector products Hq from automatic differentiation to estimate the k most negative eigenvectors v_i and Rayleigh quotients \lambda_i=v_i^ op Hv_i. Reorthogonalize the basis, sort eigenpairs by increasing eigenvalue, and reject vectors whose eigenvalue is nonnegative unless the experiment explicitly targets positive-curvature scaling. Compute c=V^ op g and choose \epsilon=\rho\,\mathrm{median}_i|\lambda_i|, with a minimum numerical floor. Form the direction d=-g+2V c+V((s-1)\odot c), where s_i=(|\lambda_i|+\epsilon)^{-1}; equivalently, d=-g+2\sum_i c_i v_i+\sum_i(s_i-1)c_i v_i. Apply heta_{t+1}= heta_t+\eta d, followed by optional norm clipping and decoupled weight decay. Pseudocode: g=grad(E,theta); pairs=Lanczos(HVP(E,theta),k); lambda,V=sort(pairs); c=V.T@g; s=1/(abs(lambda)+eps); d=-g+2*V@c+V@((s-1)*c); theta=theta+eta*d. The paper supplies the reflection and reciprocal-curvature rule; Lanczos residual tolerance, damping, learning rate, and refresh frequency are empirical choices. First test on a synthetic quadratic with eigenvalues spanning 10^6 and a known index-k saddle, then a 2-layer 256-unit CIFAR-10 MLP. Compare against ordinary gradient descent and unscaled HiSD using equal Hessian-vector-product budgets. Measure iterations and wall-clock time to reach a fixed gradient norm, loss decrease per HVP, and failure rate under larger step sizes. Success is curvature-independent convergence on the quadratic and faster saddle localization without update explosions.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a deterministic low-rank curvature-scaled HiSD update for exact quadratic saddles and saved reproducible results. The measured rate matched |lambda|/(|lambda|+epsilon) with maximum absolute error 5.5e-12; the curvature-rate ratio compressed from 10,000x for HiSD to 1.10x for the scaled method over |lambda|=0.01..100, and the observed Euler stability boundaries matched the predicted eta=4 scaled versus eta=2 ordinary boundaries on the sweep grid. In the index-2 quadratic comparison, steps to gradient norm below 1e-3 fell from 1153 to 32, while wall time was essentially unchanged/slightly slower due to the extra arithmetic, so the mechanism is real but a runtime win was not demonstrated.

Agent confidence: 9/10

Baseline: HiSD: final negative-subspace gradient norm 2.062e-4; 1153 steps to ||g||<1e-3; 0.1132 s
Idea: Curvature-scaled HiSD: final negative-subspace gradient norm 1.898e-6; 32 steps to ||g||<1e-3; 0.1238 s
Math checks: Rate max error 5.46e-12; ordinary/scaled rate ratio 10000/1.10; stability boundaries observed at 2.0 and 4.0 on a 0.05 grid

Limitations: Only exact synthetic diagonal quadratics were tested. No automatic-differentiation Hessian-vector products, Lanczos eigenvector estimation, noisy/stale curvature estimates, MLP/CIFAR experiment, GPU execution, or high-dimensional low-rank approximation was evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.03030) Sep 2, 2026

Significantly WORSE than the tuned baseline (+9.0%, p=0.0367, wins 2 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 11.6078
Idea mean: 12.6499
Effect (Δ): 1.0421 (+9.0%; negative = idea better)
Wins: 2 / 8 paired seeds
p-value: 0.0367 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±7.1%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched tabular optimizer track was tested. Larger models, other tracks, noisy or stale curvature estimates, alternative refresh frequencies, weight decay, and larger k were not tested. The final-checkpoint mechanism signature could not be quantitatively retested because no negative-curvature directions survived the Lanczos threshold.

How to run: python3 bench_stage2.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

The required bench round was rerun on the registered built-in tabular track with 8 paired seeds and a tuned Adam baseline. The matched low-rank curvature-scaled optimizer significantly underperformed: test MSE 12.649931 versus 11.607812, paired delta +1.042119, permutation p=0.0367. The NN-scale mechanism signature was unavailable because no negative-curvature Lanczos directions survived at the final checkpoints.

Baseline: Adam mlp_tiny on Friedman#1 tabular: best lr=0.01; mean test MSE 11.607812.
Comparison: delta_mean=+1.042119; idea_wins=2/8; permutation p=0.0367; verdict=idea worse (significant).
Idea: Low-rank curvature-scaled optimizer with mlp_tiny: best lr=0.01; mean test MSE 12.649931.

Benchmark result