△ Mechanism confirmed, baseline not beaten 2026

Convex Bayesian Potential Head

Implementation & benchmark of arXiv:2607.06252 — A Convex Approximation Framework for Neural Likelihood-Based Bayesian Inverse Problems

Usefulness6/10

Difficulty6/10

Novelty6/10

Optimization Stability Accuracy Generalization Vae Optimizer Loss Training dynamics Convex analysis Probability Statistics Optimization

Source paper: A Convex Approximation Framework for Neural Likelihood-Based Bayesian Inverse Problems arXiv:2607.06252 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the usual unconstrained neural likelihood head with an unnormalized posterior potential that is linear in a learned coefficient vector over neural features. Optimize the exact partition-function-corrected posterior objective rather than only pointwise negative log-likelihood. This gives a globally convex final-layer problem and a positive-semidefinite covariance Hessian, reducing optimizer sensitivity and calibration failures.

Formulas

$$\frac{d\mu^{y}}{d\mu}(x)=\frac{1}{Z(y)}\exp\left(-L(x;y)\right),\qquad Z(y)=\int_H\exp\left(-L(x;y)\right)\mu(dx).$$

$$L_\theta(x;y)=\theta^\top\phi_\psi(x,y),\qquad Z_\theta(y)=\int_H\exp\left[-\theta^\top\phi_\psi(x,y)\right]\mu(dx).$$

$$J(\theta)=\mathbb E_{(x,y)\sim\lambda}\left[\theta^\top\phi_\psi(x,y)+\log Z_\theta(y)\right].$$

$$\nabla_\theta J=\mathbb E_\lambda[\phi_\psi(x,y)]-\mathbb E_{q_\theta(x\mid y)}[\phi_\psi(x,y)],\qquad \nabla_\theta^2J=\mathbb E_y\left[\operatorname{Cov}_{q_\theta(x\mid y)}(\phi_\psi(x,y))\right]\succeq0.$$

Mathematical statement

The paper defines the posterior density with respect to a prior measure $\mu$ as $d\mu^y/d\mu(x)=Z(y)^{-1}\exp[-L(x;y)]$, where $x\in H$ is the latent or physical unknown, $y\in\mathbb R^m$ is the observation, $L$ is the negative log-likelihood/potential, and $Z(y)=\int_H e^{-L(x;y)}\,\mu(dx)$ is the normalizer. Use the adapted potential $L_\theta(x;y)=\theta^\top\phi_\psi(x,y)$, where $\phi_\psi\in\mathbb R^d$ is a neural feature vector and $\theta\in\mathbb R^d$ is the convex head parameter. Ignoring terms independent of $\theta$, minimizing expected posterior KL is equivalent to minimizing $J(\theta)=\mathbb E_{(x,y)\sim\lambda}[\theta^\top\phi_\psi(x,y)+\log Z_\theta(y)]$, with $Z_\theta(y)=\int_H e^{-\theta^\top\phi_\psi(x,y)}\,\mu(dx)$. Its gradient is $\nabla_\theta J=\mathbb E_\lambda[\phi_\psi(x,y)]-\mathbb E_{q_\theta(x\mid y)}[\phi_\psi(x,y)]$, and its Hessian is $\nabla^2_\theta J=\mathbb E_y[\operatorname{Cov}_{q_\theta(x\mid y)}(\phi_\psi(x,y))]\succeq0$. It is strictly convex on identifiable directions when no nonzero coefficient vector produces a feature combination that is constant $q_\theta(\cdot\mid y)$-almost everywhere for almost every $y$.

Implementation notes

Integrate this at the likelihood or posterior-scoring head of a conditional VAE, inverse-problem network, or amortized posterior model. Keep a neural encoder $\phi_\psi(x,y)\in\mathbb R^d$, initially frozen, and replace its ordinary scalar likelihood output with $L_\theta(x;y)=\theta^\top\phi_\psi(x,y)$. The prior sampler supplies particles $x_{b,k}\sim\mu$ for each observation $y_b$. For each batch, compute $s_{b,k}=\theta^\top\phi_\psi(x_{b,k},y_b)$, estimate the partition function by importance sampling, $\widehat{\log Z}_b=\operatorname{logsumexp}_k(-s_{b,k}+\log w_{b,k})$, and form normalized posterior weights $r_{b,k}=\operatorname{softmax}_k(-s_{b,k}+\log w_{b,k})$. Use the stochastic gradient $\widehat g=B^{-1}\sum_b[\phi_\psi(x_b,y_b)-\sum_k r_{b,k}\phi_\psi(x_{b,k},y_b)]$, followed by either Newton/conjugate-gradient or damped gradient updates on $\theta$. Estimate the Hessian-vector product with the weighted feature covariance, $\widehat H v=B^{-1}\sum_b[\sum_k r_{b,k}(\phi_{b,k}^\top v)\phi_{b,k}-(\sum_k r_{b,k}\phi_{b,k})(\sum_k r_{b,k}\phi_{b,k})^\top v]$. The convexity claim is mathematical; partition functions, posterior expectations, and effective sample size are estimated empirically. Start with a two-dimensional deblurring toy problem or a 2D nonlinear inverse map, using 32--128 prior particles per observation and a frozen MLP feature map. Compare against Adam-trained unconstrained scalar likelihood heads at equal feature dimension and equal particle count. Measure objective monotonicity, calibration of posterior credible intervals, final posterior KL or held-out negative log-likelihood, and sensitivity across random seeds. A successful first signal is substantially lower seed variance and reliable convergence to the same solution from widely different initializations; after validating the head, unfreeze $\psi$ only intermittently and retain convex optimization for $\theta$.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a finite-state Bayesian inverse-problem MVP with an exact partition function, frozen neural features, a convex linear potential head, covariance Hessian/Newton optimization, and an unconstrained neural-score baseline. The Hessian was PSD with minimum eigenvalue 0.00588, and the finite-difference Hessian-vector check matched to 2.3e-11. Across five initializations, the convex head converged to the same objective and posterior, with better held-out objective and KL and essentially zero seed variance than the neural baseline; this is a clear toy-level signal, not evidence of broad superiority.

Agent confidence: 9/10

Baseline: Unconstrained neural score: held-out objective -0.3595 +/- 0.0363, posterior KL 0.1760 +/- 0.0221, expected 90% coverage 0.9449 +/- 0.0034 across 5 seeds.
Idea: Convex potential head: held-out objective -0.5237 +/- 1.5e-13, posterior KL 0.0910 +/- 1.0e-13, expected 90% coverage 0.9616 +/- 0.0000 across 5 initializations; training objective -0.613880 in every run.

Limitations: Only a five-state one-dimensional latent toy problem was tested, with an exact finite prior rather than sampled particles. The feature map was frozen, the baseline had a different parameterization and was trained with a fixed Adam schedule, and no continuous high-dimensional inverse problem, GPU path, wall-clock comparison, intermittent feature unfreezing, or large-scale calibration study was evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.06252) Sep 4, 2026

Ahead of the baseline (−1.7%) but not statistically significant (p=0.0649, wins 7 / 8 paired seeds).

Benchmark: Bounded_energy_regression
Model: energy_potential_mlp
Paired seeds: 8
Baseline mean: 0.7746
Idea mean: 0.7615
Effect (Δ): -0.0131 (−1.7%; negative = idea better)
Wins: 7 / 8 paired seeds
p-value: 0.0649 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±1.7%
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

The registered matched task is a bounded 2-D energy regression rather than a full continuous inverse problem with Monte Carlo prior particles. The feature trunk was frozen, the prior integral used a fixed 64-point grid, calibration intervals and wall-clock/FLOP comparisons were not measured, and the observed win did not reach permutation significance with eight seeds.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the Convex Bayesian Potential Head as a separately trained normalized energy model against a matched unconstrained neural energy baseline on the registered bounded_energy_regression track. The idea achieved lower test MSE (0.76154 vs 0.77460; paired delta -0.01306; 7/8 wins), and the trained-model covariance Hessian signature was confirmed PSD (minimum eigenvalue 0.00470). However, p=0.06495 exceeded 0.05, so the result is a promising but non-significant win and worked=false under the bench rule.

Baseline: Best lr=0.01; full 8-seed test MSE 0.774600 +/- 0.027591. Sweep means: lr=0.01 -> 0.776752, lr=0.03 -> 0.851800, lr=0.08 -> 1.003566.
Idea: Best shared-grid lr=0.01; 8-seed test MSE 0.761540 +/- 0.030350. Idea sweep means: lr=0.01 -> 0.761540, lr=0.03 -> 0.763843, lr=0.08 -> 0.768748. Paired delta=-0.013060, 7/8 wins, permutation p=0.06495, verdict='no significant win'.

Benchmark result