Lanczos-triggered low-rank Newton Adam

Integrate this into the optimizer update, not into the network architecture. Keep Adam's usual first- and second-moment updates, and every K optimizer steps—start with K=100—run r-step Lanczos using Hessian-vector products from the current minibatch loss. First transform each HVP as $x\mapsto D^{-1/2}H(D^{-1/2}x)$, where $D=\operatorname{diag}(v_t)+\epsilon I$. Lanczos returns approximate eigenvectors $Q_r$ and eigenvalues $\mu_i$ of the symmetric operator $B$. Compute $a_i=q_i^\top D^{-1/2}g$, the total transformed gradient energy $E=\|D^{-1/2}g\|^2$, the flat fraction using threshold $\tau=10^{-3}$ times the running median of positive $|\mu_i|$, and the estimated condition number from the smallest and largest positive Ritz values. Pseudocode: (1) update Adam moments; (2) if step mod K is nonzero, apply the ordinary Adam step; (3) otherwise call Lanczos(B,r); (4) compute $F_{\mathrm{flat}}$ and $\widehat\kappa$; (5) if $F_{\mathrm{flat}}>0.3$ or $\widehat\kappa>100$, form the low-rank correction with $\gamma=0.1\max(\mu_i,\tau)$ and blend it with Adam using $\alpha\in[0.1,1]$; (6) clip the combined step to the Adam step norm times 2. Use only positive Ritz values in the inverse, and fall back to Adam if Lanczos fails or the predicted quadratic decrease is nonpositive. The mathematical quantities are estimated from HVPs; no full Hessian is constructed. Begin with a 2-layer SIREN or ReLU MLP fitting MNIST or a single $64\times64$ image, comparing Adam, Adam plus untriggered low-rank correction, and the triggered method at matched HVP/FLOP budgets. Log loss, gradient energy fractions, condition estimates, and number of corrections. Success means faster loss descent or lower final fitting error than Adam at equal wall-clock cost, especially on runs where Adam exhibits a plateau; a negative result is no improvement after accounting for the Lanczos overhead.