Adequacy-monitored hybrid subspace LM optimizer

1. Integration point: implement this as a PyTorch optimizer for a feedforward network or a small transformer block on a squared-error objective. Operate on a minibatch and use autograd Jacobian-vector products (JVPs) and vector-Jacobian products (VJPs), never materializing the full Jacobian. Flatten parameters into theta only conceptually; store each subspace vector as a parameter-shaped PyTree. 2. Pseudocode: compute residual r and gradient g; initialize a candidate list with g/||g||, the last m accepted updates, and optionally random Gaussian parameter vectors. For each candidate v, compute Hv=J^T(Jv) using one JVP followed by one VJP, then orthogonalize all candidates with modified Gram–Schmidt to obtain U. Compute eta=||U(U^Tg)||/||g||. If eta<tau, append Krylov vectors Hq for q equal to the current gradient or the latest basis vector, plus s randomized Hv vectors, reorthogonalize, and recompute eta. Form B=U^THU by applying H to every column of U, solve (B+lambda I)z=-U^Tg by Cholesky or eigendecomposition, and set p=Uz. Apply a line search or fixed accepted step policy, then retain the accepted p in the memory queue. 3. Computed from the mathematics: projected curvature B, LM step z, and adequacy eta. Estimated empirically: H through minibatch JVP/VJP products, with optional damping or diagonal stabilization if B is numerically indefinite. 4. First experiment: train a 2-layer 256-unit MLP on UCI or synthetic nonlinear regression with batch size 512. Compare AdamW, full-batch Gauss–Newton/LM, and a fixed-dimension Krylov LM at equal numbers of forward/backward or JVP/VJP calls. Use k=8 initially, tau=0.8, m=2 memory vectors, two Krylov vectors, and two random probes, with k_max=32. Success means faster loss reduction per wall-clock second or per curvature product, no increase in failed steps, and comparable final MSE; log eta, subspace dimension, damping, and the fraction of iterations requiring enrichment.