# Instrument-Godambe Preconditioner

- ID: 739
- Canonical URL: https://synthcore.org/idea/739/instrument-godambe-preconditioner
- API JSON: https://synthcore.org/api/idea/739.json
- API Markdown: https://synthcore.org/api/idea/739.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.11246](https://arxiv.org/abs/2607.11246)
- Category: optimization
- Solves: stability, speedup, accuracy
- ML areas: optimizer, training-dynamics, transformer
- Math tags: geometry, statistics, linear-algebra, differential-geometry
- Ratings: usefulness 7/10; difficulty 6/10; novelty 6/10

## Idea description

Build a low-dimensional neural-network geometry from trainable observables or probes instead of estimating the full Fisher matrix. Precondition the parameter gradient by the inverse variability of the probes and their parameter sensitivity, producing a task-adapted update that can remain usable for implicit models, heavy-tailed data, and parameter-dependent-support distributions.

## Mathematical statement

The paper defines the Fisher metric by I(    heta)_{jk}=\mathrm{E}_{    heta}[\partial_{    heta_j}\log p_{    heta}(X)\,\partial_{    heta_k}\log p_{    heta}(X)], which requires a dominating density p_    heta, differentiable log likelihood, and finite positive-definite information. Its weak replacement is the Godambe metric G(    heta)=S(    heta)^{    op}V(    heta)^{-1}S(    heta). Let \psi(X,    heta)\in\mathbb{R}^{m} be an arbitrary vector of weak inference instruments or neural probes, with mean-zero residuals \mathrm{E}_{    heta}[\psi(X,    heta)]=0. Define S(    heta)=\mathrm{E}_{    heta}[
abla_{    heta}\psi(X,    heta)]\in\mathbb{R}^{m    imes p}, where p is the number of trainable parameters, and V(    heta)=\mathrm{Cov}_{    heta}(\psi(X,    heta))\in\mathbb{R}^{m    imes m}. Positive-definite V and full-rank sensitivity make G positive semidefinite and quantify parameter directions that strongly affect low-variance probes. For minibatch training, estimate S by \widehat S=B^{-1}\sum_{i=1}^{B}J_i, with J_i=
abla_    heta\psi(x_i,    heta), and estimate V by \widehat V=(B-1)^{-1}\sum_i(\psi_i-\bar\psi)(\psi_i-\bar\psi)^{    op}. Use the damped metric \widehat G_\gamma=\widehat S^{    op}(\widehat V+\varepsilon I_m)^{-1}\widehat S+\gamma I_p, where \varepsilon>0 stabilizes probe covariance and \gamma>0 supplies full-rank damping. The update solves \widehat G_\gamma\,\Delta    heta=
abla_    heta L and sets     heta\leftarrow    heta-\eta\Delta    heta. The paper's key transferable property is inverse-variance weighting of arbitrary sensitivities, not the existence of a likelihood score.

## Key formulas

- $$I(\theta)_{jk}=\mathrm{E}_{\theta}\!\left[\frac{\partial\log p_{\theta}(X)}{\partial\theta_j}\frac{\partial\log p_{\theta}(X)}{\partial\theta_k}\right]$$
- $$G(\theta)=S(\theta)^{\top}V(\theta)^{-1}S(\theta)$$
- $$\widehat S=\frac{1}{B}\sum_{i=1}^{B}J_i,\qquad \widehat V=\frac{1}{B-1}\sum_{i=1}^{B}(\psi_i-\bar\psi)(\psi_i-\bar\psi)^{\top}$$
- $$\widehat G_{\gamma}=\widehat S^{\top}(\widehat V+\varepsilon I_m)^{-1}\widehat S+\gamma I_p,\qquad \widehat G_{\gamma}\Delta\theta=\nabla_{\theta}L,\qquad \theta\leftarrow\theta-\eta\Delta\theta$$

## Implementation notes

Integrate this into the optimizer update for a small Transformer or MLP, initially only for one parameter block such as the final projection or attention output projection. Construct m=8--32 probes from the current network output: for classification, let \psi(x,    heta)=R(\mathrm{logits}_{    heta}(x)-\mathrm{onehot}(y)) plus a few random projections of the penultimate hidden state, where R is a fixed m-by-d Gaussian or Rademacher matrix. Center the probes over each minibatch. For each example compute the probe Jacobian J_i=
abla_    heta\psi(x_i,    heta) using forward-mode JVPs when m is small, or one reverse-mode VJP per probe; do not materialize the p-by-p matrix. Compute \widehat V from the centered probe values, add \varepsilon I_m, and factor it with a Cholesky decomposition. Implement multiplication by the metric through v -> \widehat S v -> (\widehat V+\varepsilon I)^{-1}(\widehat S v) -> \widehat S^{    op}(\cdot)+\gamma v. Solve (\widehat G_\gamma)\Delta    heta=g with 5--20 conjugate-gradient iterations, where g is the ordinary minibatch gradient, then apply the update with gradient clipping disabled or separately ablated. Exponentially average \widehat V and \widehat S across minibatches if estimates are noisy. The mathematical quantities are the sensitivity and covariance estimates; probe dimension m, damping, EMA decay, and CG tolerance are empirical hyperparameters. First test on CIFAR-10 with a 2-layer MLP and a 4-layer small ViT against AdamW and diagonal Adam using equal wall-clock and FLOP budgets. Measure loss decrease per optimizer step, gradient-noise amplification, final accuracy, and added overhead. A useful success signal is faster loss descent or lower variance at equal training compute, especially under injected Cauchy-tailed input noise or label-shifted minibatches where Fisher-style assumptions are unreliable.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a reproducible NumPy MVP for the instrument-Godambe metric, including empirical probe covariance and sensitivity estimates, damping, and a quadratic optimizer comparison. All three mechanism predictions passed: inverse-variance geometry matched within 0.31%, the stability boundary matched the predicted |1-eta h/gamma|<=1 condition, and sensitivity scaling followed the predicted square law to numerical precision. The Godambe update reduced the ill-conditioned quadratic loss to 2.46e-38 versus 0.305 for fixed-step gradient descent after 30 steps, although this is an idealized mechanism test rather than evidence of a neural-network win.

### Mechanism check

- Verdict: Built a reproducible NumPy MVP for the instrument-Godambe metric, including empirical probe covariance and sensitivity estimates, damping, and a quadratic optimizer comparison. All three mechanism predictions passed: inverse-variance geometry matched within 0.31%, the stability boundary matched the predicted |1-eta h/gamma|<=1 condition, and sensitivity scaling followed the predicted square law to numerical precision. The Godambe update reduced the ill-conditioned quadratic loss to 2.46e-38 versus 0.305 for fixed-step gradient descent after 30 steps, although this is an idealized mechanism test rather than evidence of a neural-network win.
- Confidence: 9/10
- Limitations: No CIFAR-10, MLP or Transformer, minibatch autodiff Jacobians, CG implementation, AdamW comparison, wall-clock or FLOP accounting, heavy-tailed inputs, or label-shift experiment was tested. The toy probes are linear with known sensitivities, and the quadratic comparison uses a favorable geometry-aligned probe construction.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.9235
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_godambe.py](https://synthcore.org/code/310/bench_godambe.py)
- [bench_results.json](https://synthcore.org/code/310/bench_results.json)
- [godambe_experiment.py](https://synthcore.org/code/310/godambe_experiment.py)
- [report.md](https://synthcore.org/code/310/report.md)
- [report_bench_2026-08-31T005124.md](https://synthcore.org/code/310/report_bench_2026-08-31T005124.md)
- [results.json](https://synthcore.org/code/310/results.json)
- [Download all files as ZIP](https://synthcore.org/download/310)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.