# Convex Bayesian Potential Head

- ID: 467
- Canonical URL: https://synthcore.org/idea/467/convex-bayesian-potential-head
- API JSON: https://synthcore.org/api/idea/467.json
- API Markdown: https://synthcore.org/api/idea/467.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.06252](https://arxiv.org/abs/2607.06252)
- Category: optimization
- Solves: stability, accuracy, generalization
- ML areas: vae, optimizer, loss, training-dynamics
- Math tags: convex-analysis, probability, statistics, optimization
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace the usual unconstrained neural likelihood head with an unnormalized posterior potential that is linear in a learned coefficient vector over neural features. Optimize the exact partition-function-corrected posterior objective rather than only pointwise negative log-likelihood. This gives a globally convex final-layer problem and a positive-semidefinite covariance Hessian, reducing optimizer sensitivity and calibration failures.

## Mathematical statement

The paper defines the posterior density with respect to a prior measure \(\mu\) as \(d\mu^y/d\mu(x)=Z(y)^{-1}\exp[-L(x;y)]\), where \(x\in H\) is the latent or physical unknown, \(y\in\mathbb R^m\) is the observation, \(L\) is the negative log-likelihood/potential, and \(Z(y)=\int_H e^{-L(x;y)}\,\mu(dx)\) is the normalizer. Use the adapted potential \(L_\theta(x;y)=\theta^\top\phi_\psi(x,y)\), where \(\phi_\psi\in\mathbb R^d\) is a neural feature vector and \(\theta\in\mathbb R^d\) is the convex head parameter. Ignoring terms independent of \(\theta\), minimizing expected posterior KL is equivalent to minimizing \(J(\theta)=\mathbb E_{(x,y)\sim\lambda}[\theta^\top\phi_\psi(x,y)+\log Z_\theta(y)]\), with \(Z_\theta(y)=\int_H e^{-\theta^\top\phi_\psi(x,y)}\,\mu(dx)\). Its gradient is \(\nabla_\theta J=\mathbb E_\lambda[\phi_\psi(x,y)]-\mathbb E_{q_\theta(x\mid y)}[\phi_\psi(x,y)]\), and its Hessian is \(\nabla^2_\theta J=\mathbb E_y[\operatorname{Cov}_{q_\theta(x\mid y)}(\phi_\psi(x,y))]\succeq0\). It is strictly convex on identifiable directions when no nonzero coefficient vector produces a feature combination that is constant \(q_\theta(\cdot\mid y)\)-almost everywhere for almost every \(y\).

## Key formulas

- $$\frac{d\mu^{y}}{d\mu}(x)=\frac{1}{Z(y)}\exp\left(-L(x;y)\right),\qquad Z(y)=\int_H\exp\left(-L(x;y)\right)\mu(dx).$$
- $$L_\theta(x;y)=\theta^\top\phi_\psi(x,y),\qquad Z_\theta(y)=\int_H\exp\left[-\theta^\top\phi_\psi(x,y)\right]\mu(dx).$$
- $$J(\theta)=\mathbb E_{(x,y)\sim\lambda}\left[\theta^\top\phi_\psi(x,y)+\log Z_\theta(y)\right].$$
- $$\nabla_\theta J=\mathbb E_\lambda[\phi_\psi(x,y)]-\mathbb E_{q_\theta(x\mid y)}[\phi_\psi(x,y)],\qquad \nabla_\theta^2J=\mathbb E_y\left[\operatorname{Cov}_{q_\theta(x\mid y)}(\phi_\psi(x,y))\right]\succeq0.$$

## Implementation notes

Integrate this at the likelihood or posterior-scoring head of a conditional VAE, inverse-problem network, or amortized posterior model. Keep a neural encoder \(\phi_\psi(x,y)\in\mathbb R^d\), initially frozen, and replace its ordinary scalar likelihood output with \(L_\theta(x;y)=\theta^\top\phi_\psi(x,y)\). The prior sampler supplies particles \(x_{b,k}\sim\mu\) for each observation \(y_b\). For each batch, compute \(s_{b,k}=\theta^\top\phi_\psi(x_{b,k},y_b)\), estimate the partition function by importance sampling, \(\widehat{\log Z}_b=\operatorname{logsumexp}_k(-s_{b,k}+\log w_{b,k})\), and form normalized posterior weights \(r_{b,k}=\operatorname{softmax}_k(-s_{b,k}+\log w_{b,k})\). Use the stochastic gradient \(\widehat g=B^{-1}\sum_b[\phi_\psi(x_b,y_b)-\sum_k r_{b,k}\phi_\psi(x_{b,k},y_b)]\), followed by either Newton/conjugate-gradient or damped gradient updates on \(\theta\). Estimate the Hessian-vector product with the weighted feature covariance, \(\widehat H v=B^{-1}\sum_b[\sum_k r_{b,k}(\phi_{b,k}^\top v)\phi_{b,k}-(\sum_k r_{b,k}\phi_{b,k})(\sum_k r_{b,k}\phi_{b,k})^\top v]\). The convexity claim is mathematical; partition functions, posterior expectations, and effective sample size are estimated empirically. Start with a two-dimensional deblurring toy problem or a 2D nonlinear inverse map, using 32--128 prior particles per observation and a frozen MLP feature map. Compare against Adam-trained unconstrained scalar likelihood heads at equal feature dimension and equal particle count. Measure objective monotonicity, calibration of posterior credible intervals, final posterior KL or held-out negative log-likelihood, and sensitivity across random seeds. A successful first signal is substantially lower seed variance and reliable convergence to the same solution from widely different initializations; after validating the head, unfreeze \(\psi\) only intermittently and retain convex optimization for \(\theta\).

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a finite-state Bayesian inverse-problem MVP with an exact partition function, frozen neural features, a convex linear potential head, covariance Hessian/Newton optimization, and an unconstrained neural-score baseline. The Hessian was PSD with minimum eigenvalue 0.00588, and the finite-difference Hessian-vector check matched to 2.3e-11. Across five initializations, the convex head converged to the same objective and posterior, with better held-out objective and KL and essentially zero seed variance than the neural baseline; this is a clear toy-level signal, not evidence of broad superiority.

### Mechanism check

- Verdict: Built a finite-state Bayesian inverse-problem MVP with an exact partition function, frozen neural features, a convex linear potential head, covariance Hessian/Newton optimization, and an unconstrained neural-score baseline. The Hessian was PSD with minimum eigenvalue 0.00588, and the finite-difference Hessian-vector check matched to 2.3e-11. Across five initializations, the convex head converged to the same objective and posterior, with better held-out objective and KL and essentially zero seed variance than the neural baseline; this is a clear toy-level signal, not evidence of broad superiority.
- Confidence: 9/10
- Limitations: Only a five-state one-dimensional latent toy problem was tested, with an exact finite prior rather than sampled particles. The feature map was frozen, the baseline had a different parameterization and was trained with a fixed Adam schedule, and no continuous high-dimensional inverse problem, GPU path, wall-clock comparison, intermittent feature unfreezing, or large-scale calibration study was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.06495
- Paired wins: 7/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/1313/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/1313/bench_report.json)
- [conditional_bayesian_potential_track.py](https://synthcore.org/code/1313/conditional_bayesian_potential_track.py)
- [experiment.py](https://synthcore.org/code/1313/experiment.py)
- [report.md](https://synthcore.org/code/1313/report.md)
- [report_bench_2026-09-04T140605.md](https://synthcore.org/code/1313/report_bench_2026-09-04T140605.md)
- [results.json](https://synthcore.org/code/1313/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1313)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.