# Prescribed-Error Jeffreys Calibration

- ID: 277
- Canonical URL: https://synthcore.org/idea/277/prescribed-error-jeffreys-calibration
- API JSON: https://synthcore.org/api/idea/277.json
- API Markdown: https://synthcore.org/api/idea/277.md
- Verification status: unverified
- Source: [arXiv:2606.31593](https://arxiv.org/abs/2606.31593)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: loss, training, rlhf, world-model
- Math tags: information-theory, statistics, probability
- Ratings: usefulness 5/10; difficulty 3/10; novelty 5/10

## Idea description

Add a calibration loss that explicitly enforces independently prescribed false-positive and false-negative targets rather than optimizing symmetric accuracy or cross-entropy alone. Apply it to binary classifiers, OOD detectors, safety heads, or abstention gates, and use the binary Jeffreys divergence to encourage robust separation between null and alternative score distributions.

## Mathematical statement

The paper defines a randomized test \(\phi:\mathcal X\to[0,1]\), where \(\phi(x)\) is the probability of rejecting the null, with separate requirements \(\sup_{P\in\mathcal P_0}\int\phi\,dP\leq\alpha\) and \(\sup_{P\in\mathcal P_1}\int(1-\phi)\,dP\leq\beta\). The binary reduction preserves two directed information requirements associated with \(\alpha\) and \(\beta\). For a neural scalar score \(s=f_\theta(x)\) and threshold \(t\), let \(P_0^s,P_1^s\) be the score distributions under null and alternative data, \(p_0(t)=\Pr_{P_0}(s\geq t)\), and \(p_1(t)=\Pr_{P_1}(s\geq t)\). Then type-I error is \(p_0(t)\), and type-II error is \(1-p_1(t)\). For Bernoulli probabilities define \(d(a\|b)=a\log(a/b)+(1-a)\log((1-a)/(1-b))\). The binary Jeffreys divergence is \(J(t)=d(p_0(t)\|p_1(t))+d(p_1(t)\|p_0(t))\). The implementation minimizes cross-entropy, independent quadratic violations of the \(\alpha\) and \(\beta\) constraints, and \(-\lambda_JJ(t)\). Since empirical probabilities can equal zero or one, use Laplace smoothing \(\widetilde p=(k+1)/(n+2)\) before evaluating KL terms. The mathematical asset is directional information control: the two error targets are not collapsed into a single symmetric risk.

## Key formulas

- $$\sup_{P\in\mathcal P_0}\int\phi\,\mathrm dP\leq\alpha,\qquad \sup_{P\in\mathcal P_1}\int(1-\phi)\,\mathrm dP\leq\beta.$$
- $$p_0(t)=\Pr_{P_0}(s\geq t),\qquad p_1(t)=\Pr_{P_1}(s\geq t),\qquad \mathrm{FPR}(t)=p_0(t),\quad \mathrm{FNR}(t)=1-p_1(t).$$
- $$d(a\|b)=a\log\frac{a}{b}+(1-a)\log\frac{1-a}{1-b},\qquad J(t)=d(p_0(t)\|p_1(t))+d(p_1(t)\|p_0(t)).$$
- $$\mathcal L(\theta,t)=\mathcal L_{\mathrm{CE}}(\theta)+\lambda_0[\max(0,\widehat{\mathrm{FPR}}(t)-\alpha)]^2+\lambda_1[\max(0,\widehat{\mathrm{FNR}}(t)-\beta)]^2-\lambda_JJ(t).$$

## Implementation notes

(1) Integration point: use the scalar logit of a binary classifier, OOD detector, reward-model safety head, or abstention gate. Keep cross-entropy for representation learning and add this objective during a calibration fine-tuning stage; treat the decision threshold \(t\) as an explicit post-training parameter rather than modifying hidden layers. (2) Pseudocode: split data into training, calibration, and test sets. Train \(f_\theta\) with cross-entropy. On each calibration batch or over the full calibration set, compute scores, choose a threshold \(t\), and count \(k_0=\sum_{x\in D_0}\mathbf 1[s_\theta(x)\geq t]\) and \(k_1=\sum_{x\in D_1}\mathbf 1[s_\theta(x)\geq t]\). Set \(\widetilde p_0=(k_0+1)/(n_0+2)\), \(\widetilde p_1=(k_1+1)/(n_1+2)\), then compute FPR, FNR, both KL terms, and the displayed loss. For differentiable training, replace the indicator by \(\sigma((s-t)/\tau)\), annealing \(\tau\); for final deployment, scan sorted validation scores. Select the smallest threshold whose upper-confidence FPR is at most \(\alpha\) and whose upper-confidence FNR is at most \(\beta\), using Clopper-Pearson or binomial-KL confidence bounds. (3) The paper supplies the separate-error and directed-divergence construction; probabilities, threshold, and confidence bounds are estimated empirically. (4) First experiment: train a small ResNet on CIFAR-10 versus an OOD split, or an MLP on a binary safety dataset. Compare cross-entropy with ordinary threshold tuning against this method at \((\alpha,\beta)=(10^{-2},10^{-2})\), \((10^{-3},10^{-2})\), and \((10^{-2},10^{-3})\). On a fresh test set measure worst-group FPR/FNR, constraint violations, AUROC, and calibration-set size. Success is fewer violations of both prescribed constraints at equal model size and accuracy, particularly when the targets are asymmetric, with negligible inference overhead.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.