Instrument-Godambe Preconditioner

Integrate this into the optimizer update for a small Transformer or MLP, initially only for one parameter block such as the final projection or attention output projection. Construct m=8--32 probes from the current network output: for classification, let \psi(x, heta)=R(\mathrm{logits}_{ heta}(x)-\mathrm{onehot}(y)) plus a few random projections of the penultimate hidden state, where R is a fixed m-by-d Gaussian or Rademacher matrix. Center the probes over each minibatch. For each example compute the probe Jacobian J_i=
abla_ heta\psi(x_i, heta) using forward-mode JVPs when m is small, or one reverse-mode VJP per probe; do not materialize the p-by-p matrix. Compute \widehat V from the centered probe values, add \varepsilon I_m, and factor it with a Cholesky decomposition. Implement multiplication by the metric through v -> \widehat S v -> (\widehat V+\varepsilon I)^{-1}(\widehat S v) -> \widehat S^{ op}(\cdot)+\gamma v. Solve (\widehat G_\gamma)\Delta heta=g with 5--20 conjugate-gradient iterations, where g is the ordinary minibatch gradient, then apply the update with gradient clipping disabled or separately ablated. Exponentially average \widehat V and \widehat S across minibatches if estimates are noisy. The mathematical quantities are the sensitivity and covariance estimates; probe dimension m, damping, EMA decay, and CG tolerance are empirical hyperparameters. First test on CIFAR-10 with a 2-layer MLP and a 4-layer small ViT against AdamW and diagonal Adam using equal wall-clock and FLOP budgets. Measure loss decrease per optimizer step, gradient-noise amplification, final accuracy, and added overhead. A useful success signal is faster loss descent or lower variance at equal training compute, especially under injected Cauchy-tailed input noise or label-shifted minibatches where Fisher-style assumptions are unreliable.