✓✓ Beats tuned baseline 2026

Covariance Fisher Preconditioner

Implementation & benchmark of arXiv:2607.04578 — A Differentiable Covariance Calculus for Linear Gaussian Bayesian Networks

Usefulness7/10

Difficulty6/10

Novelty5/10

Optimization Stability Speedup Sample efficiency Optimizer Vae Rnn Training dynamics Information theory Statistics Linear algebra Optimization

Source paper: A Differentiable Covariance Calculus for Linear Gaussian Bayesian Networks arXiv:2607.04578 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the differentiable covariance chart to construct a Fisher-information preconditioner for the edge and innovation parameters of a linear-Gaussian neural module. Instead of applying an isotropic Euclidean update, whiten parameter steps according to how strongly they change the predicted Gaussian distribution. This targets ill-conditioning caused by redundant paths, correlated latent nodes, and badly scaled innovation covariances.

Formulas

$$C=(I-A)^{-1}D(I-A)^{-\mathsf T},\qquad S=PCP^{\mathsf T}.$$

$$F_{ab}=\frac{1}{2}\operatorname{tr}\!\left(S^{-1}\frac{\partial S}{\partial\theta_a}S^{-1}\frac{\partial S}{\partial\theta_b}\right).$$

$$\theta_{t+1}=\theta_t-\eta\left(F(\theta_t)+\lambda I\right)^{-1}g_t,\qquad g_t=\nabla_\theta\mathcal{L}(\theta_t).$$

Mathematical statement

For a zero-mean Gaussian observation y distributed as N(0,S(theta)), the covariance-only Slepian-Bangs Fisher information has entries F_ab = 1/2 tr(S^(-1)(partial S/partial theta_a) S^(-1)(partial S/partial theta_b)). Here theta_a and theta_b are scalar trainable edge or innovation parameters, S is an observed covariance, and S may be obtained from the full covariance C=(I-A)^(-1)D(I-A)^(-T) by S=P C P^T, where P selects observed nodes or features. The Fisher matrix is positive semidefinite because it is a Gram matrix under the inner product <X,Y>_S = 1/2 tr(S^(-1) X S^(-1) Y). Use a stabilized inverse (S+delta I)^(-1), with delta>0, and a damped natural-gradient-like update (F+lambda I)^(-1)g, where g is the neural loss gradient and lambda>0 prevents singular solves. For large parameter sets, use only diagonal F_aa or low-rank/Hutchinson approximations.

Implementation notes

Integrate this into the optimizer for parameters belonging to the Gaussian DAG layer, while leaving the rest of the network on AdamW. Every K optimizer steps, construct S=P@C@P.T from the current layer parameters, add delta*I, and compute g=grad(loss,theta) for the selected parameter vector theta. For an MVP with 20-200 scalar parameters, use autodiff to obtain J_a=partial S/partial theta_a. Compute W=inv(S+delta*I), then F[a,b]=0.5*trace(W@J_a@W@J_b), and update theta with solve(F+lambda*I,g). Compact pseudocode: C=covariance_chart(A,Sigma); S=P@C@P.T+delta*I; J=jacobian(S,theta); W=cholesky_solve(S,I); F[a,b]=0.5*trace(W@J[a]@W@J[b]); step=solve(F+lambda*I,g); theta=theta-eta*step. Derivatives are obtained by reverse- or forward-mode autodiff; lambda, delta, and update interval K are tuned empirically. For scaling, replace F by its diagonal F_aa=0.5*trace(W@J_a@W@J_a), or estimate quadratic forms with Hutchinson probes. First test an 8-16 dimensional latent structured VAE or Kalman-like recurrent predictor with deliberately rescaled edge matrices and correlated Sigma_j. Compare AdamW, AdaBelief, and this preconditioner at equal function evaluations. Track loss versus steps, wall-clock, gradient variance, covariance condition number, and divergence frequency. Success is faster held-out NLL reduction or fewer unstable runs at comparable compute.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an autodiff covariance-chart Fisher preconditioner for a small linear-Gaussian DAG and compared it with explicit AdamW. The mechanism manifested: all 12 PSD checks passed, uniform covariance scaling matched the predicted invariant Fisher norm with maximum absolute error 6.0e-6, and observed damped contractions matched theory within 1.4e-16. The optimizer effect was not a consistent win: Fisher was better at the largest learning rate but worse early or essentially tied otherwise, while costing 4.1–4.5 seconds versus 0.69–0.87 seconds for AdamW.

Agent confidence: 9/10

Baseline: AdamW final NLL: 1.69519, 1.69522, 1.69530 at learning rates 0.08, 0.16, 0.04; NLL at step 20: 1.75348, 1.83129, 2.27296; runtime: 0.69–0.87 s.
Idea: Fisher final NLL: 1.69518, 1.69518, 1.69896 at learning rates 0.08, 0.16, 0.04; NLL at step 20: 1.95585, 1.71198, 2.46970; runtime: 4.10–4.46 s. PSD pass fraction 12/12; scale-invariance max error 6.0e-6; contraction max error 1.4e-16.

Limitations: Only a tiny 5-parameter, 4-node Gaussian DAG and 100-step deterministic toy comparison were tested. No VAE, larger parameter vector, diagonal/low-rank Fisher approximation, held-out dataset, multiple random training seeds, FLOP-normalized comparison, or comparison with AdaBelief was evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.04578) Sep 2, 2026

Beats the tuned baseline by 48.1% (p=0.0163, wins 7 / 8 paired seeds; metric: lower is better).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0028
Idea mean: 0.0015
Effect (Δ): -0.0013 (−48.1%; negative = idea better)
Wins: 7 / 8 paired seeds
p-value: 0.0163 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±33.4%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 6 configs

Limitations:

Only the registered dynamics track, 8 paired seeds, 10 epochs, and a diagonal covariance Fisher approximation over the recurrent output head were tested. Full-matrix Fisher, larger latent Gaussian modules, FLOP-normalized timing, and AdaBelief were not tested.

How to run: OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 CUDA_VISIBLE_DEVICES='' /home/maxwelhelp/main/bin/python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Worked ✓

Agent confidence: 8/10

The covariance-Fisher intervention was tested on the registered dynamics track with the matched rnn_small architecture, tuned Adam baseline, shared hyperparameter grid, and 8 paired seeds. It significantly improved test MSE: delta_mean=-0.0013460118 with permutation p=0.0163 and 7/8 wins. The trained-model covariance-scale signature confirmed the predicted Fisher invariance exactly, although only a diagonal head approximation was tested.

Baseline: Tuned Adam best configuration lr=0.006, weight_decay=0.0001; mean test MSE=0.0027980784, std=0.0009924122 over 8 seeds.
Idea: Diagonal covariance-Fisher preconditioner best configuration lr=0.006, weight_decay=0.0; mean test MSE=0.0014520666, std=0.0004280286 over 8 seeds.

Benchmark result