# Amortized low-rank Laplace hyperparameter marginalization

- ID: 394
- Canonical URL: https://synthcore.org/idea/394/amortized-low-rank-laplace-hyperparameter-marginalization
- API JSON: https://synthcore.org/api/idea/394.json
- API Markdown: https://synthcore.org/api/idea/394.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.03355](https://arxiv.org/abs/2607.03355)
- Category: optimization
- Solves: speedup, scalability, memory
- ML areas: optimizer, fine-tuning, regularization, memory
- Math tags: linear-algebra, approximation-theory, probability
- Ratings: usefulness 6/10; difficulty 5/10; novelty 5/10

## Idea description

Replace repeated full-curvature factorizations when evaluating many Bayesian neural-network hyperparameter settings with one shared low-rank approximation of the data-induced precision update. Reuse the same curvature subspace while changing prior precision, observation-noise precision, or layerwise regularization, evaluating each candidate through a small rank-r matrix and Woodbury solves.

## Mathematical statement

The paper starts from the linear conditional-Gaussian model ${\boldsymbol y}=Am+{\boldsymbol\varepsilon}$, where $m$ is the latent parameter, $A$ is the parameter-to-observable map, and ${\boldsymbol\varepsilon}$ is observation noise. For a neural-network Laplace approximation, let $w\in\mathbb{R}^{d}$ be parameters, $w_\theta^*$ a MAP point for hyperparameters $\theta$, $J\in\mathbb{R}^{q\times d}$ the Jacobian of model outputs with respect to $w$, and $R_\theta\in\mathbb{R}^{q\times q}$ the observation-noise covariance. With prior precision $P_\theta\succ0$, the local posterior precision is $H_\theta=P_\theta+J^\top R_\theta^{-1}J$. The transferable generalized low-rank approximation is to represent the update $J^\top R_\theta^{-1}J$ as $U\Lambda_\theta U^\top$, with $U\in\mathbb{R}^{d\times r}$ shared across hyperparameters and $r\ll d$. The determinant lemma gives $\log\det H_\theta=\log\det P_\theta+\log\det(I_r+\Lambda_\theta^{1/2}U^\top P_\theta^{-1}U\Lambda_\theta^{1/2})$. Woodbury gives $H_\theta^{-1}=P_\theta^{-1}-P_\theta^{-1}U(\Lambda_\theta^{-1}+U^\top P_\theta^{-1}U)^{-1}U^\top P_\theta^{-1}$. These formulas turn each marginal-likelihood or quantity-of-interest evaluation into operations involving an $r\times r$ matrix, while the expensive curvature subspace is amortized over all $\theta$.

## Key formulas

- $${\boldsymbol y}=Am+{\boldsymbol\varepsilon}.$$
- $$H_\theta=P_\theta+J^\top R_\theta^{-1}J\;\approx\;P_\theta+U\Lambda_\theta U^\top,$$
- $$\log\det H_\theta=\log\det P_\theta+\log\det\!\left(I_r+\Lambda_\theta^{1/2}U^\top P_\theta^{-1}U\Lambda_\theta^{1/2}\right),$$
- $$H_\theta^{-1}=P_\theta^{-1}-P_\theta^{-1}U\left(\Lambda_\theta^{-1}+U^\top P_\theta^{-1}U\right)^{-1}U^\top P_\theta^{-1}.$$

## Implementation notes

(1) Integrate at the Laplace-evidence or Bayesian fine-tuning component, immediately after obtaining a MAP network $w^*$ and before evaluating many candidate hyperparameters. Use the network-output Jacobian or generalized Gauss--Newton curvature; do not form a dense $d\times d$ Hessian. Let $P_\theta$ encode weight decay, for example diagonal or layerwise scalar precisions, and let $R_\theta=\sigma_\theta^2I$ encode output-noise hyperparameters. (2) Pseudocode: draw a calibration batch $B$; compute Jacobian-vector products for $J_B$; estimate the top $r$ eigenpairs of $J_B^\top R_0^{-1}J_B$ using randomized Lanczos or randomized range finding; orthonormalize the resulting vectors to obtain $U$ and store the projected update coefficients $\Lambda_\theta$ for each candidate noise setting. For every candidate $\theta$, compute $S_\theta=\Lambda_\theta^{-1}+U^\top P_\theta^{-1}U$, evaluate $\log\det H_\theta$ with the determinant formula, and solve posterior systems with the Woodbury formula. If the prior is diagonal, all applications of $P_\theta^{-1}$ are elementwise. (3) The paper-derived mathematical operations are the shared low-rank precision update, determinant lemma, and Woodbury inverse. Estimate $U$ empirically from minibatch Jacobian-vector products; estimate truncation quality by monitoring captured Rayleigh energy $\operatorname{tr}(U^\top J^\top J U)/\operatorname{tr}(J^\top J)$ and held-out quadratic-form errors. Recompute $U$ only after a large MAP change or when this error exceeds a threshold. (4) First experiment: use a small MLP or ViT-tiny on MNIST or CIFAR-10, fit a MAP model, and compare 32--128 weight-decay/noise candidates. Benchmark exact per-candidate generalized Gauss--Newton Cholesky on the final layer against ranks $r\in\{16,32,64\}$. Measure evidence ranking agreement, predictive negative log-likelihood, wall-clock time, peak memory, and marginal-quantity agreement. Success means at least 3x lower evaluation time and 2x lower peak memory at rank 64 while preserving evidence ranking or test NLL within 1% of the exact baseline.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a NumPy MVP implementing the shared low-rank precision update, determinant lemma, and Woodbury solve. The full-rank formulas matched exact Cholesky and solve results to machine precision; at rank 64, candidate evaluation was 7.85x faster with perfect hyperparameter ranking agreement and 96.6% captured trace energy. The approximation was not within 1% for every quantity: relative log-determinant RMSE was 3.94% and worst quadratic-form error was 1.61%, so the speed and ranking effect is real but accuracy is only moderate.

### Mechanism check

- Verdict: Built a NumPy MVP implementing the shared low-rank precision update, determinant lemma, and Woodbury solve. The full-rank formulas matched exact Cholesky and solve results to machine precision; at rank 64, candidate evaluation was 7.85x faster with perfect hyperparameter ranking agreement and 96.6% captured trace energy. The approximation was not within 1% for every quantity: relative log-determinant RMSE was 3.94% and worst quadratic-form error was 1.61%, so the speed and ranking effect is real but accuracy is only moderate.
- Confidence: 8/10
- Limitations: This is a synthetic linearized-curvature benchmark, not a neural-network Laplace/MAP experiment. The low-rank eigenspace was computed with a dense eigendecomposition, so subspace-construction cost and repeated-MAP effects were not benchmarked; memory, GPU behavior, randomized Lanczos, layerwise priors, and larger models were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.10015
- Paired wins: 6/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/1285/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/1285/bench_report.json)
- [experiment.py](https://synthcore.org/code/1285/experiment.py)
- [report.md](https://synthcore.org/code/1285/report.md)
- [report_bench_2026-09-04T115138.md](https://synthcore.org/code/1285/report_bench_2026-09-04T115138.md)
- [results.json](https://synthcore.org/code/1285/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1285)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.