# Single-Krylov Hessian Spectral Oracle

- ID: 321
- Canonical URL: https://synthcore.org/idea/321/single-krylov-hessian-spectral-oracle
- API JSON: https://synthcore.org/api/idea/321.json
- API Markdown: https://synthcore.org/api/idea/321.md
- Verification status: unverified
- Source: [arXiv:2607.01114](https://arxiv.org/abs/2607.01114)
- Category: optimization
- Solves: speedup, stability
- ML areas: optimizer, training-dynamics
- Math tags: spectral-theory, linear-algebra, numerical-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Use Hessian-vector products and one Lanczos run to approximate the loss-curvature spectral density and resolvent response over many damping values. Feed the estimated spectral edges and Lorentzian mass into an adaptive optimizer that selects learning-rate and damping parameters, avoiding repeated frequency-by-frequency curvature solves.

## Mathematical statement

The paper rewrites the response as a quadratic form of a resolvent. In its notation, A and B are QRPA blocks, F is the external-field vector, omega_gamma is the queried frequency with positive broadening gamma, and U diagonalizes the QRPA operator with eigenvalue matrix Omega; equation (24) gives S(omega_gamma)=-(U^dagger F)^dagger diag((Omega-omega_gamma I)^(-1),(Omega+omega_gamma I)^(-1))(U^dagger F). Its imaginary part is a sum of Lorentzians centered at eigenvalues, gamma/(pi((omega-lambda)^2+gamma^2)). For a neural network, let H=nabla_theta^2 L(theta) be the symmetric Hessian, q a normalized random probe, and z=omega+i gamma; the target response is r(z)=q^T(H-zI)^(-1)q. Lanczos constructs orthonormal vectors v_j and a tridiagonal matrix T_k using alpha_j=v_j^T H v_j and beta_(j+1)=||H v_j-alpha_j v_j-beta_j v_(j-1)||_2. The Krylov approximation is r(z) approximately e_1^T(T_k-zI)^(-1)e_1, valid for every z after one run. The broadened empirical spectral density is rho_gamma(omega)=-Im(e_1^T(T_k-(omega+i gamma)I)^(-1)e_1)/pi.

## Key formulas

- $$S(\omega_{\gamma})=-\left(\mathcal{U}^{\dagger}\mathcal{F}\right)^{\dagger}\begin{bmatrix}(\Omega-\omega_{\gamma}I)^{-1}&0\\0&(\Omega+\omega_{\gamma}I)^{-1}\end{bmatrix}\left(\mathcal{U}^{\dagger}\mathcal{F}\right)$$
- $$-\frac{1}{\pi}\operatorname{Im}S(\omega_{\gamma})=\sum_i |c_i|^2\frac{\gamma/\pi}{(\omega-\lambda_i)^2+\gamma^2}$$
- $$r(z)=q^{\mathsf T}(H-zI)^{-1}q\ \approx\ e_1^{\mathsf T}(T_k-zI)^{-1}e_1,\qquad \rho_\gamma(\omega)=-\frac{1}{\pi}\operatorname{Im}e_1^{\mathsf T}\left(T_k-(\omega+i\gamma)I\right)^{-1}e_1$$
- $$Hv_j=\beta_jv_{j-1}+\alpha_jv_j+\beta_{j+1}v_{j+1},\quad \alpha_j=v_j^{\mathsf T}Hv_j,\quad \beta_{j+1}=\left\|Hv_j-\alpha_jv_j-\beta_jv_{j-1}\right\|_2$$

## Implementation notes

(1) Integration point: add a curvature-monitoring and hyperparameter-control branch to an AdamW or SGD training loop. It operates on the flattened trainable parameter vector theta, using Hessian-vector products from Pearlmutter autodiff; it must not materialize the Hessian. Every M optimizer steps, choose a random Gaussian probe q, normalize it, and run k=20 to 50 Lanczos iterations. (2) Pseudocode: initialize v_0=0, v_1=q, and beta_1=0. For j=1,...,k, compute u=HVP(L,theta,v_j), alpha_j=v_j^T u, then replace u by u-alpha_j v_j-beta_j v_(j-1). With full or selective reorthogonalization, remove numerical components along previous vectors. Set beta_(j+1)=||u||_2 and, if nonzero, set v_(j+1)=u/beta_(j+1). Store the tridiagonal T_k with diagonal alpha_j and off-diagonal beta_(j+1). (3) Evaluate rho_gamma(omega) for a grid of omega and damping values by eigendecomposing the small T_k or solving small shifted systems. Set the optimizer learning-rate ceiling to eta_t=c/(max(lambda_i(T_k),epsilon)), where c is a safety factor such as 0.5 to 1.0 and epsilon prevents division by zero. Set an additional damping or trust-region parameter from the smallest gamma whose resolvent response is numerically stable. (4) Computed from the paper's mathematics are the resolvent approximation, Lorentzian spectral density, and reuse of one Krylov basis for all query frequencies. Estimated empirically are HVPs, stochastic probe coverage, spectral-edge accuracy, and the constants c and epsilon; use two to eight independent probes and average their densities if variance is high. (5) First experiment: train a small six-layer Transformer or ResNet on CIFAR-10 or a one-billion-token language-model slice, comparing AdamW with cosine decay against AdamW plus the Lanczos schedule at equal training FLOPs. Run k=30 every 200 steps and measure HVP overhead separately. Success means fewer divergence events, faster loss decrease at equal optimizer steps, or matching final loss with a larger stable learning rate. Validate the oracle by comparing its top eigenvalue with occasional power-iteration measurements and require relative error below 10 to 20 percent.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.