Local Irreducible-Vertex Preconditioner

Integrate this as an optimizer for small MLPs or attention projection layers, initially using one block per layer or parameter tensor. Maintain exponential moving averages \(\chi_b\leftarrow\beta\chi_b+(1-\beta)g_bg_b^\top\) and a bare estimate \(\chi_{0,b}\) from K-FAC factors or running marginal variances. Every 50–200 optimizer steps, compute Cholesky- or eigenvalue-ridged inverses, form \(\Gamma_b=\chi_{0,b}^{-1}-\chi_b^{-1}\), clip the eigenvalues of \(\chi_{0,b}^{-1}-\Gamma_b\) to at least \(\lambda\), and apply \(\Delta\theta_b=-\eta P_bg_b\). Use minibatch subsampling and low-rank covariance updates to control cost. The paper supplies inverse-susceptibility subtraction and the locality assumption for the fully irreducible interaction; gradient covariances, block structure, and approximation error are estimated empirically. First test on a 3-layer MNIST MLP and a small transformer encoder, comparing Adam, K-FAC, and this local-vertex optimizer at equal matrix-operation budgets. Measure held-out covariance reconstruction and loss decrease per update. The prediction is that the locality error \(\varepsilon_{\mathrm{local}}\) should fall below 0.1–0.2 when ladder-like correlations dominate; otherwise the locality assumption should be rejected. Stable learning-rate gain should persist until the largest eigenvalue of the preconditioned local Hessian approaches the quadratic-update boundary near two.