✗ Failed on benchmark 2026

Amortized low-rank Laplace hyperparameter marginalization

Implementation & benchmark of arXiv:2607.03355 — Amortized low-rank approximation for hyperparameter marginalization in PDE-governed Bayesian inverse problems

Usefulness6/10

Difficulty5/10

Novelty5/10

Optimization Speedup Scalability Memory Optimizer Fine tuning Regularization Memory Linear algebra Approximation theory Probability

Source paper: Amortized low-rank approximation for hyperparameter marginalization in PDE-governed Bayesian inverse problems arXiv:2607.03355 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace repeated full-curvature factorizations when evaluating many Bayesian neural-network hyperparameter settings with one shared low-rank approximation of the data-induced precision update. Reuse the same curvature subspace while changing prior precision, observation-noise precision, or layerwise regularization, evaluating each candidate through a small rank-r matrix and Woodbury solves.

Formulas

$${\boldsymbol y}=Am+{\boldsymbol\varepsilon}.$$

$$H_\theta=P_\theta+J^\top R_\theta^{-1}J\;\approx\;P_\theta+U\Lambda_\theta U^\top,$$

$$\log\det H_\theta=\log\det P_\theta+\log\det\!\left(I_r+\Lambda_\theta^{1/2}U^\top P_\theta^{-1}U\Lambda_\theta^{1/2}\right),$$

$$H_\theta^{-1}=P_\theta^{-1}-P_\theta^{-1}U\left(\Lambda_\theta^{-1}+U^\top P_\theta^{-1}U\right)^{-1}U^\top P_\theta^{-1}.$$

Mathematical statement

The paper starts from the linear conditional-Gaussian model ${\boldsymbol y}=Am+{\boldsymbol\varepsilon}$, where $m$ is the latent parameter, $A$ is the parameter-to-observable map, and ${\boldsymbol\varepsilon}$ is observation noise. For a neural-network Laplace approximation, let $w\in\mathbb{R}^{d}$ be parameters, $w_\theta^*$ a MAP point for hyperparameters $\theta$, $J\in\mathbb{R}^{q\times d}$ the Jacobian of model outputs with respect to $w$, and $R_\theta\in\mathbb{R}^{q\times q}$ the observation-noise covariance. With prior precision $P_\theta\succ0$, the local posterior precision is $H_\theta=P_\theta+J^\top R_\theta^{-1}J$. The transferable generalized low-rank approximation is to represent the update $J^\top R_\theta^{-1}J$ as $U\Lambda_\theta U^\top$, with $U\in\mathbb{R}^{d\times r}$ shared across hyperparameters and $r\ll d$. The determinant lemma gives $\log\det H_\theta=\log\det P_\theta+\log\det(I_r+\Lambda_\theta^{1/2}U^\top P_\theta^{-1}U\Lambda_\theta^{1/2})$. Woodbury gives $H_\theta^{-1}=P_\theta^{-1}-P_\theta^{-1}U(\Lambda_\theta^{-1}+U^\top P_\theta^{-1}U)^{-1}U^\top P_\theta^{-1}$. These formulas turn each marginal-likelihood or quantity-of-interest evaluation into operations involving an $r\times r$ matrix, while the expensive curvature subspace is amortized over all $\theta$.

Implementation notes

(1) Integrate at the Laplace-evidence or Bayesian fine-tuning component, immediately after obtaining a MAP network $w^*$ and before evaluating many candidate hyperparameters. Use the network-output Jacobian or generalized Gauss--Newton curvature; do not form a dense $d\times d$ Hessian. Let $P_\theta$ encode weight decay, for example diagonal or layerwise scalar precisions, and let $R_\theta=\sigma_\theta^2I$ encode output-noise hyperparameters. (2) Pseudocode: draw a calibration batch $B$; compute Jacobian-vector products for $J_B$; estimate the top $r$ eigenpairs of $J_B^\top R_0^{-1}J_B$ using randomized Lanczos or randomized range finding; orthonormalize the resulting vectors to obtain $U$ and store the projected update coefficients $\Lambda_\theta$ for each candidate noise setting. For every candidate $\theta$, compute $S_\theta=\Lambda_\theta^{-1}+U^\top P_\theta^{-1}U$, evaluate $\log\det H_\theta$ with the determinant formula, and solve posterior systems with the Woodbury formula. If the prior is diagonal, all applications of $P_\theta^{-1}$ are elementwise. (3) The paper-derived mathematical operations are the shared low-rank precision update, determinant lemma, and Woodbury inverse. Estimate $U$ empirically from minibatch Jacobian-vector products; estimate truncation quality by monitoring captured Rayleigh energy $\operatorname{tr}(U^\top J^\top J U)/\operatorname{tr}(J^\top J)$ and held-out quadratic-form errors. Recompute $U$ only after a large MAP change or when this error exceeds a threshold. (4) First experiment: use a small MLP or ViT-tiny on MNIST or CIFAR-10, fit a MAP model, and compare 32--128 weight-decay/noise candidates. Benchmark exact per-candidate generalized Gauss--Newton Cholesky on the final layer against ranks $r\in\{16,32,64\}$. Measure evidence ranking agreement, predictive negative log-likelihood, wall-clock time, peak memory, and marginal-quantity agreement. Success means at least 3x lower evaluation time and 2x lower peak memory at rank 64 while preserving evidence ranking or test NLL within 1% of the exact baseline.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a NumPy MVP implementing the shared low-rank precision update, determinant lemma, and Woodbury solve. The full-rank formulas matched exact Cholesky and solve results to machine precision; at rank 64, candidate evaluation was 7.85x faster with perfect hyperparameter ranking agreement and 96.6% captured trace energy. The approximation was not within 1% for every quantity: relative log-determinant RMSE was 3.94% and worst quadratic-form error was 1.61%, so the speed and ranking effect is real but accuracy is only moderate.

Agent confidence: 8/10

Baseline: 64 exact dense Cholesky evaluations: 30.86 sec; exact ranking used as reference.
Idea: Rank-64 shared Woodbury evaluations: 3.93 sec, 7.85x speedup, Spearman ranking correlation 1.000, captured trace fraction 0.966, relative logdet RMSE 0.0394, worst relative quadratic-form error 0.0161.

Limitations: This is a synthetic linearized-curvature benchmark, not a neural-network Laplace/MAP experiment. The low-rank eigenspace was computed with a dense eigendecomposition, so subspace-construction cost and repeated-MAP effects were not benchmarked; memory, GPU behavior, randomized Lanczos, layerwise priors, and larger models were not tested.

How to run: python3 experiment.py > results.json

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.03355) Sep 4, 2026

Ahead of the baseline (−8.3%) but not statistically significant (p=0.1002, wins 6 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 9.7748
Idea mean: 8.9637
Effect (Δ): -0.8111 (−8.3%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.1002 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the tabular/Friedman#1 track and mlp_tiny architecture were tested. The intervention was an optimizer-level Woodbury preconditioner rather than full Laplace evidence or repeated neural-network hyperparameter marginal-likelihood evaluation; wall-clock and memory savings for candidate evidence evaluation were not measured. The low-rank subspace used an exact small calibration SVD rather than randomized Lanczos, and only rank 16 was evaluated.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented an end-to-end tabular MLP variant using a shared rank-16 trained-model Jacobian subspace and Woodbury-style gradient preconditioning, with identical architecture, data, epochs, and hyperparameter grid as the Adam baseline. Mean test MSE improved from 9.774816 to 8.963684, but the paired permutation p-value was 0.10015, so the mandatory harness verdict is no significant win. The honest trained-model signature found only 21.3% mean held-out Jacobian energy captured by the rank-16 subspace, so the proposed transfer mechanism was not confirmed.

Baseline: Tabular/Friedman#1 mlp_tiny, Adam, best lr=0.006 and weight_decay=0.0001; 8-seed mean test MSE 9.774816, std 1.146524.
Delta mean: -0.811133 MSE (-8.30% relative)
Idea: Shared rank-16 Jacobian Woodbury preconditioning, best lr=0.006 and weight_decay=0.0001; 8-seed mean test MSE 8.963684, std 1.218498.
Idea wins: 6/8 paired seeds
Mechanism signature confirmed: false
Mechanism signature mean captured fraction: 0.21334
Permutation p value: 0.10015

Benchmark result