# Covariance Fisher Preconditioner

- ID: 409
- Canonical URL: https://synthcore.org/idea/409/covariance-fisher-preconditioner
- API JSON: https://synthcore.org/api/idea/409.json
- API Markdown: https://synthcore.org/api/idea/409.md
- Verification status: beats_baseline
- Source: [arXiv:2607.04578](https://arxiv.org/abs/2607.04578)
- Category: optimization
- Solves: stability, speedup, sample-efficiency
- ML areas: optimizer, vae, rnn, training-dynamics
- Math tags: information-theory, statistics, linear-algebra, optimization
- Ratings: usefulness 7/10; difficulty 6/10; novelty 5/10

## Idea description

Use the differentiable covariance chart to construct a Fisher-information preconditioner for the edge and innovation parameters of a linear-Gaussian neural module. Instead of applying an isotropic Euclidean update, whiten parameter steps according to how strongly they change the predicted Gaussian distribution. This targets ill-conditioning caused by redundant paths, correlated latent nodes, and badly scaled innovation covariances.

## Mathematical statement

For a zero-mean Gaussian observation y distributed as N(0,S(theta)), the covariance-only Slepian-Bangs Fisher information has entries F_ab = 1/2 tr(S^(-1)(partial S/partial theta_a) S^(-1)(partial S/partial theta_b)). Here theta_a and theta_b are scalar trainable edge or innovation parameters, S is an observed covariance, and S may be obtained from the full covariance C=(I-A)^(-1)D(I-A)^(-T) by S=P C P^T, where P selects observed nodes or features. The Fisher matrix is positive semidefinite because it is a Gram matrix under the inner product <X,Y>_S = 1/2 tr(S^(-1) X S^(-1) Y). Use a stabilized inverse (S+delta I)^(-1), with delta>0, and a damped natural-gradient-like update (F+lambda I)^(-1)g, where g is the neural loss gradient and lambda>0 prevents singular solves. For large parameter sets, use only diagonal F_aa or low-rank/Hutchinson approximations.

## Key formulas

- $$C=(I-A)^{-1}D(I-A)^{-\mathsf T},\qquad S=PCP^{\mathsf T}.$$
- $$F_{ab}=\frac{1}{2}\operatorname{tr}\!\left(S^{-1}\frac{\partial S}{\partial\theta_a}S^{-1}\frac{\partial S}{\partial\theta_b}\right).$$
- $$\theta_{t+1}=\theta_t-\eta\left(F(\theta_t)+\lambda I\right)^{-1}g_t,\qquad g_t=\nabla_\theta\mathcal{L}(\theta_t).$$

## Implementation notes

Integrate this into the optimizer for parameters belonging to the Gaussian DAG layer, while leaving the rest of the network on AdamW. Every K optimizer steps, construct S=P@C@P.T from the current layer parameters, add delta*I, and compute g=grad(loss,theta) for the selected parameter vector theta. For an MVP with 20-200 scalar parameters, use autodiff to obtain J_a=partial S/partial theta_a. Compute W=inv(S+delta*I), then F[a,b]=0.5*trace(W@J_a@W@J_b), and update theta with solve(F+lambda*I,g). Compact pseudocode: C=covariance_chart(A,Sigma); S=P@C@P.T+delta*I; J=jacobian(S,theta); W=cholesky_solve(S,I); F[a,b]=0.5*trace(W@J[a]@W@J[b]); step=solve(F+lambda*I,g); theta=theta-eta*step. Derivatives are obtained by reverse- or forward-mode autodiff; lambda, delta, and update interval K are tuned empirically. For scaling, replace F by its diagonal F_aa=0.5*trace(W@J_a@W@J_a), or estimate quadratic forms with Hutchinson probes. First test an 8-16 dimensional latent structured VAE or Kalman-like recurrent predictor with deliberately rescaled edge matrices and correlated Sigma_j. Compare AdamW, AdaBelief, and this preconditioner at equal function evaluations. Track loss versus steps, wall-clock, gradient variance, covariance condition number, and divergence frequency. Success is faster held-out NLL reduction or fewer unstable runs at comparable compute.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built an autodiff covariance-chart Fisher preconditioner for a small linear-Gaussian DAG and compared it with explicit AdamW. The mechanism manifested: all 12 PSD checks passed, uniform covariance scaling matched the predicted invariant Fisher norm with maximum absolute error 6.0e-6, and observed damped contractions matched theory within 1.4e-16. The optimizer effect was not a consistent win: Fisher was better at the largest learning rate but worse early or essentially tied otherwise, while costing 4.1–4.5 seconds versus 0.69–0.87 seconds for AdamW.

### Mechanism check

- Verdict: Built an autodiff covariance-chart Fisher preconditioner for a small linear-Gaussian DAG and compared it with explicit AdamW. The mechanism manifested: all 12 PSD checks passed, uniform covariance scaling matched the predicted invariant Fisher norm with maximum absolute error 6.0e-6, and observed damped contractions matched theory within 1.4e-16. The optimizer effect was not a consistent win: Fisher was better at the largest learning rate but worse early or essentially tied otherwise, while costing 4.1–4.5 seconds versus 0.69–0.87 seconds for AdamW.
- Confidence: 9/10
- Limitations: Only a tiny 5-parameter, 4-node Gaussian DAG and 100-step deterministic toy comparison were tested. No VAE, larger parameter vector, diagonal/low-rank Fisher approximation, held-out dataset, multiple random training seeds, FLOP-normalized comparison, or comparison with AdaBelief was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0163
- Paired wins: 7/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/208/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/208/bench_report.json)
- [bench_run.py](https://synthcore.org/code/208/bench_run.py)
- [experiment.py](https://synthcore.org/code/208/experiment.py)
- [report.md](https://synthcore.org/code/208/report.md)
- [report_bench_2026-08-30T205542.md](https://synthcore.org/code/208/report_bench_2026-08-30T205542.md)
- [results.json](https://synthcore.org/code/208/results.json)
- [Download all files as ZIP](https://synthcore.org/download/208)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.