# Cramer-energy embedding confidence

- ID: 2858
- Canonical URL: https://synthcore.org/idea/2858/cramer-energy-embedding-confidence
- API JSON: https://synthcore.org/api/idea/2858.json
- API Markdown: https://synthcore.org/api/idea/2858.md
- Verification status: unverified
- Source: [arXiv:2608.29972](https://arxiv.org/abs/2608.29972)
- Category: regularization
- Solves: accuracy, generalization, sample-efficiency
- ML areas: embedding, loss, curriculum
- Math tags: probability, convex-analysis, geometry
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Use the Legendre-dual Cramér transform of the minibatch embedding distribution as a geometry-aware novelty and confidence score. Add a penalty that separates correctly classified embeddings from high-Cramér-energy examples, or use the energy directly for OOD detection and hard-example mining; unlike squared distance, it adapts to anisotropic and non-Gaussian convex support.

## Mathematical statement

For a probability measure \(\mu\) on \(\mathbb{R}^d\), the logarithmic Laplace transform is \(\Lambda_\mu(\theta)=\log\int e^{\langle\theta,z\rangle}\,d\mu(z)\), where \(z\) is an embedding and \(\theta\in\mathbb{R}^d\) is a dual variable. Its Cramér transform is \(\Lambda_\mu^*(x)=\sup_{\theta\in\mathbb{R}^d}\{\langle\theta,x\rangle-\Lambda_\mu(\theta)\}\), where \(x\) is a candidate embedding. For the uniform measure \(\mu_K\) on a convex body \(K\), the paper proves \(\Lambda_K^*(x)\le -\log q_K(x)\le \Lambda_K^*(x)+\tfrac12\log d+C\), where \(q_K(x)=\inf\{\mu_K(H):H\text{ is a closed half-space containing }x\}\), \(d\) is embedding dimension, and \(C\) is an absolute constant. Thus high \(\Lambda^*\) corresponds to low half-space depth. Estimate the transform from a minibatch using \(\widehat\Lambda(\theta)=\log\frac1B\sum_{i=1}^B\exp(\langle\theta,z_i\rangle)\), and compute the regularized finite-sample energy \(E(x)=\max_{\theta\in\Theta}\{\langle\theta,x\rangle-\widehat\Lambda(\theta)-\tfrac{\tau^2}{2}\|\theta\|_2^2\}\), with \(\tau>0\) stabilizing dual optimization.

## Key formulas

- $$\Lambda_\mu(\theta)=\log\int_{\mathbb{R}^{n}}e^{\langle\theta,x\rangle}\,d\mu(x)$$
- $$\Lambda_\mu^{*}(x)=\sup_{\theta\in\mathbb{R}^{n}}\left\{\langle\theta,x\rangle-\Lambda_\mu(\theta)\right\}$$
- $$\Lambda_{K}^{*}(x)\leq-\log q_{K}(x)\leq\Lambda_{K}^{*}(x)+\frac{1}{2}\log n+C$$
- $$E(x)=\max_{\theta\in\mathbb{R}^{d}}\left\{\langle\theta,x\rangle-\log\left(\frac{1}{B}\sum_{i=1}^{B}e^{\langle\theta,z_i\rangle}\right)-\frac{\tau^{2}}{2}\|\theta\|_2^2\right\}$$

## Implementation notes

Integrate this at the penultimate embedding tensor of a classifier or encoder. First project embeddings to dimension \(d\leq128\), detach a running memory bank of \(B\) recent in-distribution embeddings, and whiten them with an exponential-moving-average mean and diagonal covariance; clip each coordinate to a bounded interval so the empirical support is approximately convex and numerically controlled. For every queried embedding \(x\), solve the regularized dual problem by 5–10 gradient-ascent steps on \(\theta\), initialized at zero or at the previous step’s solution. The gradient is \(x-\sum_i w_i z_i-\tau^2\theta\), with \(w_i=\operatorname{softmax}_i(\langle\theta,z_i\rangle)\). Compute \(E(x)\) from the final dual objective. Pseudocode: `theta=0; repeat: logits=Z@theta; w=softmax(logits); g=x-w.T@Z-tau**2*theta; theta=theta+eta*g; E=x@theta-logsumexp(logits)+log(B)-0.5*tau**2*sum(theta**2)`. Use \(E(x)\) as an OOD score, or add \(\lambda E(z)\) to incorrectly classified or low-margin examples. The mathematical quantities are the dual objective and depth interpretation; the memory-bank distribution, whitening statistics, \(\tau\), step size, and threshold are estimated empirically. Test a ResNet-18 on CIFAR-10 against Mahalanobis distance and standard energy scoring, using CIFAR-100 or SVHN as OOD data. Success means higher AUROC or AUPR at matched false-positive rate, with no more than 10% training overhead.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.