Sequential Sketched Gauss-Newton

Integrate this into a neural Gauss-Newton or Levenberg-Marquardt optimizer for a parameter-efficient adapter, last layer, or low-rank LoRA block. At the beginning of an optimizer step, freeze the base network, compute current predictions and residuals \(r\), and represent Jacobian-vector and transposed-Jacobian-vector products using autodiff rather than materializing \(J\). Choose row-sketch sizes such as \(m_k\in\{0.05N,0.15N,0.4N,N\}\), using either uniformly sampled examples without replacement or a CountSketch. At each level, define the operator \(A_kv=J^\top S_k^\top S_k(Jv)\); run 3-20 PCG iterations from the previous level's \(\delta\), with diagonal preconditioner \(M_k=\operatorname{diag}(A_k)+\lambda I\). Stop early when the sketched residual norm falls below a relative tolerance, then pass the resulting \(\delta_k\) to the next level. On the final level evaluate full-data residuals and run a small fixed number of refinement iterations, optionally using damping \(A_K+\lambda I\). The exact mathematical quantities are the sketched least-squares objective, warm-start recursion, and PCG update; empirically estimate diagonal preconditioners from Hutchinson probes or accumulated squared Jacobian-vector products. First test on a frozen ResNet-18 or small Transformer with a trainable linear head and compare against full-batch CG and Adam at equal wall-clock time. Measure loss decrease per Jacobian-vector product, total examples processed, final validation accuracy, and whether the sequential method reaches the same validation loss with fewer full-data passes. A useful success signal is at least 1.5x lower wall-clock cost or 2x fewer full-batch Jacobian products at matched accuracy.