Core-Response Gram Preconditioner

(1) Integration point: implement this as an optimizer preconditioner immediately before the parameter update. Flatten the trainable parameters into p in R^d. Let g=gradient_p of the minibatch loss. Use Adam's bias-corrected diagonal second moment, a diagonal Gauss-Newton estimate, or a block-diagonal K-FAC estimate as the positive-definite core M; add damping epsilon I so the inverse exists. Construct U in R^(b by d) from per-example gradients u_i=gradient_p loss_i, optionally scaled by 1/sqrt(b), with one row per example. (2) Pseudocode: M_inv = inverse_core(M); Y = U @ M_inv; G = Y @ U.T; compute the top-r eigendecomposition G V_r = V_r Lambda_r using symmetric Lanczos; C = M_inv @ U.T @ V_r; q = V_r.T @ U @ M_inv @ g; precond_g = M_inv @ g - C @ (q / (1 + Lambda_r)); update p <- p - eta * precond_g, with optional decoupled weight decay. The Woodbury equation supplies the exact correction. (3) Compute from the paper's mathematics: G, its leading eigenvectors and eigenvalues, and the low-rank inverse correction. Estimate empirically: M, damping, rank r, and the refresh interval. Refresh the eigenspace every 10 to 100 steps and reuse it between refreshes. To reduce memory, apply M_inv layerwise and store only r preconditioned interaction vectors rather than a dense d by d matrix. For large batches, form U from a random microbatch or use a small reservoir of per-example gradients. (4) First cheap experiment: train a two-layer MLP and a small ViT on MNIST and CIFAR-10 using SGD, AdamW, diagonal natural-gradient/K-FAC, and this method at ranks r in {1,2,4,8}. Match wall-clock time and gradient FLOPs, and measure loss after a fixed update budget, steps to target accuracy, final accuracy, and sensitivity to learning rate. Compare response-Gram selection against raw-gradient PCA and random interaction directions. Success means faster loss descent or fewer steps to target accuracy at matched compute; failure means eigendecomposition and per-example-gradient overhead increases without improving optimization.