# Projective Compact Prior for Latent Models

- ID: 1014
- Canonical URL: https://synthcore.org/idea/1014/projective-compact-prior-for-latent-models
- API JSON: https://synthcore.org/api/idea/1014.json
- API Markdown: https://synthcore.org/api/idea/1014.md
- Verification status: unverified
- Source: [arXiv:2607.16547](https://arxiv.org/abs/2607.16547)
- Category: architecture
- Solves: stability, generalization, memory
- ML areas: vae, regularization, embedding
- Math tags: information-theory, probability, statistics, geometry
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Use the calibrated compact-support maximum-entropy law as a latent prior or representation regularizer in a VAE or autoencoder. Unlike a Gaussian prior, it prevents latent codes from drifting arbitrarily far while retaining explicitly controlled mean and covariance.

## Mathematical statement

Let \(z\in\mathbb R^d\) be a latent vector, \(\mu\) its target mean, and \(\Sigma\succ0\) its target covariance. Define \(r^2(z)=(z-\mu)^\top\Sigma^{-1}(z-\mu)\). For support \(r^2\le R^2\), with \(R^2>d+2\), the paper gives \(\gamma_R=2/(R^2-d-2)>0\). The compact q-Gaussian prior is \(p_R(z)=C[1-r^2(z)/R^2]_+^{1/\gamma_R}\). Its in-support negative log-prior is \(-\log C-(1/\gamma_R)\log(1-r^2/R^2)\), which diverges at the boundary and is infinite outside. The construction is affine-equivariant: under \(z' = Az+b\), \(\mu'=A\mu+b\), and \(\Sigma'=A\Sigma A^\top\), the radius \(r^2\) is unchanged.

## Key formulas

- $$\gamma_R=\frac{2}{R^2-d-2},\qquad R^2>d+2.$$
- $$r^2(z)=(z-\mu)^\top\Sigma^{-1}(z-\mu),\qquad p_R(z)=C\left[1-\frac{r^2(z)}{R^2}\right]^{1/\gamma_R}.$$
- $$\mathcal L_{\mathrm{prior}}(z)=-\log C-\frac1{\gamma_R}\log\left(1-\frac{r^2(z)}{R^2}\right),\qquad r^2(z)<R^2.$$
- $$z_{\mathrm{proj}}=\mu+\frac{R-\varepsilon}{\max(R-\varepsilon,\sqrt{r^2(z)})}(z-\mu).$$

## Implementation notes

Modify the latent regularization term of a VAE or deterministic autoencoder. Let the encoder output \(q_\phi(z\mid x)=\mathcal N(m_\phi(x),\operatorname{diag}(s_\phi^2(x)))\), or use sampled codes directly for an autoencoder. Maintain running estimates of latent mean \(\mu\) and covariance \(\Sigma\) with exponential averaging, then apply a Cholesky solve to compute \(r^2(z)\). Set a compactness radius, for example \(R^2=2(d+2)\), giving \(\gamma_R=1/(d+2)\); tune \(R\) only on validation reconstruction versus prior tradeoffs. Add \(\beta\mathcal L_{\mathrm{prior}}\) to reconstruction or task loss. Since hard infinity creates unstable gradients, use the differentiable barrier \(-\log(\max(1-r^2/R^2,\varepsilon))\) and an outside penalty \(\tau\operatorname{softplus}((r^2-R^2)/\tau)\). Optionally project sampled codes using the displayed radial projection before decoding. Pseudocode: sample \(z\); update running \(\mu,\Sigma\); compute \(r^2\), \(\gamma_R\), and the barrier; penalize or project boundary violations; backpropagate reconstruction plus prior loss. The mathematical quantities are the calibrated deformation, Mahalanobis geometry, and compact q-Gaussian; running statistics, \(\beta\), and smoothing are empirical. First test a 2-layer VAE on MNIST or CIFAR-10 against an isotropic Gaussian KL prior. Compare reconstruction, held-out likelihood, latent-radius histograms, and robustness under corruptions. Success is bounded latent norms, fewer decoder failures from extrapolative latent samples, and equal or better reconstruction at the same latent dimension.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.