# PSD-plus-low-rank curvature optimizer

- ID: 3025
- Canonical URL: https://synthcore.org/idea/3025/psd-plus-low-rank-curvature-optimizer
- API JSON: https://synthcore.org/api/idea/3025.json
- API Markdown: https://synthcore.org/api/idea/3025.md
- Verification status: mechanism_failed
- Source: [arXiv:2609.01857](https://arxiv.org/abs/2609.01857)
- Category: optimization
- Solves: stability, speedup
- ML areas: optimizer, training-dynamics, regularization
- Math tags: optimization, linear-algebra, spectral-theory, dynamical-systems
- Ratings: usefulness 8/10; difficulty 5/10; novelty 5/10

## Idea description

Approximate the minibatch loss Hessian by a positive-semidefinite bulk curvature plus a small signed transverse correction, and treat only the correction with explicit negative-curvature steps. This imports the paper's observation that all unstable directions can be confined to a low-dimensional subspace, producing a curvature-aware optimizer whose step-size boundary is governed by a small matrix rather than the full Hessian.

## Mathematical statement

The paper writes the constrained Hessian as $H_{\mathcal C}=B^{T}(D_{\mathcal P}^{2}F)B=L^{\Delta}+\widetilde L$, where $B$ embeds realizable pair coordinates into ambient pair space, $L^{\Delta}\succeq0$ is the gap Laplacian, and $\widetilde L$ is the signed transverse Laplacian. For planar four-body configurations, $\operatorname{rank}(\widetilde L)\le 2$, so negative curvature is determined by at most two generalized eigenvalues of a small symmetric pencil. In a neural network, let $g=\nabla_\theta \ell(\theta)$, let $P\succeq0$ be a diagonal, block-diagonal, or low-rank-plus-diagonal estimate of the positive bulk, and let $U\in\mathbb R^{d\times r}$ contain $r$ estimated residual curvature directions with $r\ll d$. Use $\widehat H=P+UCU^{T}$, where $C=U^{T}(\widehat H-P)U$ is an $r\times r$ symmetric signed matrix. For a locally positive quadratic model, the preconditioned update is stable when $\eta<2/\lambda_{\max}(P^{-1/2}\widehat H P^{-1/2})$.

## Key formulas

- $$H_{\mathcal C}=B^{T}(D_{\mathcal P}^{2}F)B=L^{\Delta}+\widetilde L,\qquad L^{\Delta}\succeq0,\qquad \operatorname{rank}(\widetilde L)\le 2.$$
- $$\widehat H=P+UCU^{T},\qquad C=U^{T}(\widehat H-P)U,\qquad U^{T}U=I_r.$$
- $$\eta_c=\frac{2}{\lambda_{\max}\!\left(P^{-1/2}\widehat H P^{-1/2}\right)}.$$
- $$\Delta\theta=-\eta P^{-1}g-\eta U\,\phi(C,U^{T}P^{-1}g),\qquad \phi(C,z)=\operatorname{diag}\!\left(\frac{1}{1+\gamma\min(\lambda_i(C),0)}\right)z.$$

## Implementation notes

Integrate this into AdamW or SGD immediately before the parameter update. Maintain a cheap diagonal or block-diagonal positive-semidefinite curvature estimate $P$ using exponential moving averages of squared gradients, Hutchinson diagonal Hessian estimates, or Gauss-Newton blocks. Every $K=20$ to $100$ steps, estimate the bottom $r=2$ or $4$ eigenvectors of the residual Hessian $H-P$ with two or three Lanczos iterations using Hessian-vector products; orthonormalize them to form $U$, and compute the small matrix $C$ from residual Hessian-vector products. Pseudocode: compute $g$ and $P^{-1}g$; periodically run Lanczos on $v\mapsto Hv-Pv$; set $C_{ij}=u_i^T(Hu_j-Pu_j)$; diagonalize $C$; estimate the largest eigenvalue of $M=P^{-1/2}\widehat H P^{-1/2}$; clip the global step to $0.9(2/\lambda_{\max}(M))$; optionally damp only eigenvectors associated with negative eigenvalues of $C$. The paper supplies the positive-semidefinite-plus-signed decomposition and low-rank instability principle; $P$, $U$, and $C$ are estimated empirically in the network. First experiment: train a 4-layer MLP on MNIST with SGD, AdamW, and this optimizer over a fixed learning-rate sweep. Log the first divergence boundary and the number of negative residual eigenvalues. The quantitative prediction is that the observed largest stable learning rate will be within 20% of $2/\lambda_{\max}(M)$, and that the residual negative spectrum will be concentrated in at most $r$ directions. Increasing $r$ from $2$ to $8$ should provide little additional stability if the low-rank mechanism transfers.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a PSD-plus-low-rank curvature optimizer with EMA diagonal PSD curvature, minibatch Hessian-vector products, rank-2 residual estimation, signed correction, and step clipping. The quadratic sanity check reproduced the predicted stability boundary: spectral radius was 0.90 at 0.95ηc and 1.10 at 1.05ηc, with one negative residual eigenvalue while the full Hessian remained PSD. On the fixed-seed digits MLP, however, the proposed method underperformed SGD substantially, so no neural-training win was observed.

### Mechanism check

- Verdict: Built a PSD-plus-low-rank curvature optimizer with EMA diagonal PSD curvature, minibatch Hessian-vector products, rank-2 residual estimation, signed correction, and step clipping. The quadratic sanity check reproduced the predicted stability boundary: spectral radius was 0.90 at 0.95ηc and 1.10 at 1.05ηc, with one negative residual eigenvalue while the full Hessian remained PSD. On the fixed-seed digits MLP, however, the proposed method underperformed SGD substantially, so no neural-training win was observed.
- Confidence: 9/10
- Limitations: Only one small digits dataset, one nominal learning rate, one rank, one seed, and 90 update steps were tested; no learning-rate sweep or FLOP-normalized comparison was performed. The residual curvature was estimated with tiny randomized subspace iteration on minibatch Hessian-vector products, and the neural method was not tuned against SGD.

## Artifacts

- [psd_lowrank_experiment.py](https://synthcore.org/code/1208/psd_lowrank_experiment.py)
- [report.md](https://synthcore.org/code/1208/report.md)
- [results.json](https://synthcore.org/code/1208/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1208)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.