PSD-plus-low-rank curvature optimizer
Implementation & benchmark of arXiv:2609.01857 — The Hessian of Planar Central Configurations in Pair Space: Decomposition, Morse Index and Symmetry Reduction
Source paper: The Hessian of Planar Central Configurations in Pair Space: Decomposition, Morse Index and Symmetry Reduction arXiv:2609.01857 ⓘ · analyzed Sep 3, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Approximate the minibatch loss Hessian by a positive-semidefinite bulk curvature plus a small signed transverse correction, and treat only the correction with explicit negative-curvature steps. This imports the paper's observation that all unstable directions can be confined to a low-dimensional subspace, producing a curvature-aware optimizer whose step-size boundary is governed by a small matrix rather than the full Hessian.
Formulas
Mathematical statement
The paper writes the constrained Hessian as $H_{\mathcal C}=B^{T}(D_{\mathcal P}^{2}F)B=L^{\Delta}+\widetilde L$, where $B$ embeds realizable pair coordinates into ambient pair space, $L^{\Delta}\succeq0$ is the gap Laplacian, and $\widetilde L$ is the signed transverse Laplacian. For planar four-body configurations, $\operatorname{rank}(\widetilde L)\le 2$, so negative curvature is determined by at most two generalized eigenvalues of a small symmetric pencil. In a neural network, let $g=\nabla_\theta \ell(\theta)$, let $P\succeq0$ be a diagonal, block-diagonal, or low-rank-plus-diagonal estimate of the positive bulk, and let $U\in\mathbb R^{d\times r}$ contain $r$ estimated residual curvature directions with $r\ll d$. Use $\widehat H=P+UCU^{T}$, where $C=U^{T}(\widehat H-P)U$ is an $r\times r$ symmetric signed matrix. For a locally positive quadratic model, the preconditioned update is stable when $\eta<2/\lambda_{\max}(P^{-1/2}\widehat H P^{-1/2})$.
Implementation notes
Integrate this into AdamW or SGD immediately before the parameter update. Maintain a cheap diagonal or block-diagonal positive-semidefinite curvature estimate $P$ using exponential moving averages of squared gradients, Hutchinson diagonal Hessian estimates, or Gauss-Newton blocks. Every $K=20$ to $100$ steps, estimate the bottom $r=2$ or $4$ eigenvectors of the residual Hessian $H-P$ with two or three Lanczos iterations using Hessian-vector products; orthonormalize them to form $U$, and compute the small matrix $C$ from residual Hessian-vector products. Pseudocode: compute $g$ and $P^{-1}g$; periodically run Lanczos on $v\mapsto Hv-Pv$; set $C_{ij}=u_i^T(Hu_j-Pu_j)$; diagonalize $C$; estimate the largest eigenvalue of $M=P^{-1/2}\widehat H P^{-1/2}$; clip the global step to $0.9(2/\lambda_{\max}(M))$; optionally damp only eigenvectors associated with negative eigenvalues of $C$. The paper supplies the positive-semidefinite-plus-signed decomposition and low-rank instability principle; $P$, $U$, and $C$ are estimated empirically in the network. First experiment: train a 4-layer MLP on MNIST with SGD, AdamW, and this optimizer over a fixed learning-rate sweep. Log the first divergence boundary and the number of negative residual eigenvalues. The quantitative prediction is that the observed largest stable learning rate will be within 20% of $2/\lambda_{\max}(M)$, and that the residual negative spectrum will be concentrated in at most $r$ directions. Increasing $r$ from $2$ to $8$ should provide little additional stability if the low-rank mechanism transfers.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 9/10
Built a PSD-plus-low-rank curvature optimizer with EMA diagonal PSD curvature, minibatch Hessian-vector products, rank-2 residual estimation, signed correction, and step clipping. The quadratic sanity check reproduced the predicted stability boundary: spectral radius was 0.90 at 0.95ηc and 1.10 at 1.05ηc, with one negative residual eigenvalue while the full Hessian remained PSD. On the fixed-seed digits MLP, however, the proposed method underperformed SGD substantially, so no neural-training win was observed.
- Agent confidence
- 9/10
- Baseline
- SGD, lr=0.12, 90 steps: final loss 1.5128, accuracy 56.22%, diverged=false
- Idea
- PSD-low-rank, nominal lr=0.12 with internal clipping, rank=2 and refresh every 15 steps: final loss 2.9691, minimum loss 1.6280, accuracy 15.78%, diverged=false
- Toy
- ηc=1.4569; spectral radius 0.90 below the boundary and 1.10 above it; one negative residual eigenvalue
Limitations: Only one small digits dataset, one nominal learning rate, one rank, one seed, and 90 update steps were tested; no learning-rate sweep or FLOP-normalized comparison was performed. The residual curvature was estimated with tiny randomized subspace iteration on minibatch Hessian-vector products, and the neural method was not tuned against SGD.
How to run: python3 psd_lowrank_experiment.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- psd_lowrank_experiment.py 6.8 KB View Raw
- report.md 1.5 KB View
- results.json 0.7 KB View Raw