Krylov Block-Cubic Optimizer

Integrate this at the optimizer-update point, operating on one parameter tensor or a small group of tensors at a time. For each block b, compute its gradient g_b and provide an automatic-differentiation Hessian-vector-product routine v -> H_b v; never materialize H_b. Build a k-step Lanczos basis with q_1=g_b/(||g_b||+epsilon). At iteration j, compute w=H_b q_j, set alpha_j=q_j^T w, subtract alpha_j q_j and the previous beta_j q_{j-1}, reorthogonalize if necessary, and set beta_{j+1}=||w||. Stop after k iterations or when beta_{j+1} is negligible. Store Q_k and the tridiagonal T_k. Solve the small cubic problem using the stationarity parameter lambda: for a trial lambda >= 0, solve (T_k+lambda I)z=-Q_k^T g_b and seek lambda=(sigma_b/2)||z|| by safeguarded bisection or Newton iteration. Form s_b=Q_k z and reshape it to the block tensor. Evaluate the actual loss after the proposed update. Accept only if the loss decreases; compute rho_b=[f(theta)-f(theta+s_b)]/[-m_b(s_b)], multiply sigma_b by 2 after rejection or rho_b<0.1, and halve sigma_b after rho_b>0.75, with fixed minimum and maximum bounds. For stochastic training, use a fixed validation mini-batch for the guard or periodically use full-batch checks to avoid noise-driven rejection. The paper supplies the cubic model and Krylov construction; k, block size, sigma bounds, and guard frequency are empirical choices. First test a small full-batch 2-4 layer MLP on synthetic or MNIST regression against AdamW and L-BFGS at matched wall-clock and HVP budgets. Measure loss versus time, number of HVPs, accepted-step ratio, and final error. A successful result is faster descent to a fixed loss, monotone guarded training, or better final accuracy when the Hessian contains substantial negative or anisotropic curvature.