# Low-Rank Curvature-Scaled Saddle Optimizer

- ID: 383
- Canonical URL: https://synthcore.org/idea/383/low-rank-curvature-scaled-saddle-optimizer
- API JSON: https://synthcore.org/api/idea/383.json
- API Markdown: https://synthcore.org/api/idea/383.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.03030](https://arxiv.org/abs/2607.03030)
- Category: optimization
- Solves: speedup, stability
- ML areas: optimizer, training-dynamics
- Math tags: optimization, dynamical-systems, linear-algebra, spectral-theory
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Replace the sign-flip-only dynamics of high-index saddle search with low-rank inverse-curvature scaling on the estimated negative-curvature subspace. Directions with small negative Hessian eigenvalues then receive approximately curvature-independent updates instead of extremely slow updates proportional to their tiny curvature.

## Mathematical statement

Equation (1) defines high-index saddle dynamics for an energy E(x): x is the parameter vector, g=
abla E(x), H=
abla^2E(x), v_1,\ldots,v_k are orthonormal approximations to the k unstable Hessian eigenvectors, and \beta,\gamma>0 are relaxation parameters. The position update is \dot{x}=-\beta(I-2\sum_{i=1}^k v_i v_i^    op)g, which reflects the gradient on selected eigendirections. Let \lambda_i=v_i^    op H v_i<0 be the Rayleigh curvature of v_i. The transferred method applies a damped inverse curvature to each selected direction: the orthogonal complement uses ordinary descent, while direction v_i uses multiplier (|\lambda_i|+\epsilon)^{-1}. Here V=[v_1,\ldots,v_k] has orthonormal columns and \epsilon>0 is a damping floor. For an exact eigenvector, the linearized coordinate rate becomes -\beta|\lambda_i|/(|\lambda_i|+\epsilon), so the rate no longer vanishes as |\lambda_i| becomes small compared with the other negative curvatures.

## Key formulas

- $$\left\{\begin{aligned} \frac{\mathrm{d}x}{\mathrm{d}t}&=-\beta\left(I-\sum_{i=1}^{k}2v_{i}v_{i}^{\top}\right)\nabla E(x),\\ \frac{\mathrm{d}v_{i}}{\mathrm{d}t}&=-\gamma\left(I-v_{i}v_{i}^{\top}-\sum_{j=1}^{i-1}2v_{j}v_{j}^{\top}\right)\nabla^{2}E(x)v_{i},\qquad 1\leqslant i\leqslant k.\end{aligned}\right.$$
- $$\lambda_i=v_i^\top H v_i,\qquad H=\nabla^2E(x),\qquad V=[v_1,\ldots,v_k],\qquad V^\top V=I_k.$$
- $$\dot{x}_{\mathrm{SCS}}=-\beta(I-VV^\top)g+\beta\sum_{i=1}^{k}\frac{1}{|\lambda_i|+\epsilon}v_i v_i^\top g,\qquad g=\nabla E(x).$$
- $$Hv_i=\lambda_i v_i,\ \lambda_i<0\quad\Longrightarrow\quad \dot{a}_i=-\beta\frac{|\lambda_i|}{|\lambda_i|+\epsilon}a_i,\qquad x-x^*=a_i v_i.$$

## Implementation notes

Integrate this at the parameter-update stage of a saddle-search optimizer, initially on a small MLP or transformer block with scalar objective E(    heta). At iteration t, compute g=
abla_    heta E(    heta_t). Use k-step Lanczos or LOBPCG with Hessian-vector products Hq from automatic differentiation to estimate the k most negative eigenvectors v_i and Rayleigh quotients \lambda_i=v_i^    op Hv_i. Reorthogonalize the basis, sort eigenpairs by increasing eigenvalue, and reject vectors whose eigenvalue is nonnegative unless the experiment explicitly targets positive-curvature scaling. Compute c=V^    op g and choose \epsilon=\rho\,\mathrm{median}_i|\lambda_i|, with a minimum numerical floor. Form the direction d=-g+2V c+V((s-1)\odot c), where s_i=(|\lambda_i|+\epsilon)^{-1}; equivalently, d=-g+2\sum_i c_i v_i+\sum_i(s_i-1)c_i v_i. Apply     heta_{t+1}=    heta_t+\eta d, followed by optional norm clipping and decoupled weight decay. Pseudocode: g=grad(E,theta); pairs=Lanczos(HVP(E,theta),k); lambda,V=sort(pairs); c=V.T@g; s=1/(abs(lambda)+eps); d=-g+2*V@c+V@((s-1)*c); theta=theta+eta*d. The paper supplies the reflection and reciprocal-curvature rule; Lanczos residual tolerance, damping, learning rate, and refresh frequency are empirical choices. First test on a synthetic quadratic with eigenvalues spanning 10^6 and a known index-k saddle, then a 2-layer 256-unit CIFAR-10 MLP. Compare against ordinary gradient descent and unscaled HiSD using equal Hessian-vector-product budgets. Measure iterations and wall-clock time to reach a fixed gradient norm, loss decrease per HVP, and failure rate under larger step sizes. Success is curvature-independent convergence on the quadratic and faster saddle localization without update explosions.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a deterministic low-rank curvature-scaled HiSD update for exact quadratic saddles and saved reproducible results. The measured rate matched |lambda|/(|lambda|+epsilon) with maximum absolute error 5.5e-12; the curvature-rate ratio compressed from 10,000x for HiSD to 1.10x for the scaled method over |lambda|=0.01..100, and the observed Euler stability boundaries matched the predicted eta=4 scaled versus eta=2 ordinary boundaries on the sweep grid. In the index-2 quadratic comparison, steps to gradient norm below 1e-3 fell from 1153 to 32, while wall time was essentially unchanged/slightly slower due to the extra arithmetic, so the mechanism is real but a runtime win was not demonstrated.

### Mechanism check

- Verdict: Built a deterministic low-rank curvature-scaled HiSD update for exact quadratic saddles and saved reproducible results. The measured rate matched |lambda|/(|lambda|+epsilon) with maximum absolute error 5.5e-12; the curvature-rate ratio compressed from 10,000x for HiSD to 1.10x for the scaled method over |lambda|=0.01..100, and the observed Euler stability boundaries matched the predicted eta=4 scaled versus eta=2 ordinary boundaries on the sweep grid. In the index-2 quadratic comparison, steps to gradient norm below 1e-3 fell from 1153 to 32, while wall time was essentially unchanged/slightly slower due to the extra arithmetic, so the mechanism is real but a runtime win was not demonstrated.
- Confidence: 9/10
- Limitations: Only exact synthetic diagonal quadratics were tested. No automatic-differentiation Hessian-vector products, Lanczos eigenvector estimation, noisy/stale curvature estimates, MLP/CIFAR experiment, GPU execution, or high-dimensional low-rank approximation was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0367
- Paired wins: 2/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/201/bench_report.json)
- [bench_stage2.py](https://synthcore.org/code/201/bench_stage2.py)
- [experiment.py](https://synthcore.org/code/201/experiment.py)
- [report.md](https://synthcore.org/code/201/report.md)
- [report_bench_2026-08-30T200220.md](https://synthcore.org/code/201/report_bench_2026-08-30T200220.md)
- [results.json](https://synthcore.org/code/201/results.json)
- [Download all files as ZIP](https://synthcore.org/download/201)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.