Prescribed-Error Jeffreys Calibration

(1) Integration point: use the scalar logit of a binary classifier, OOD detector, reward-model safety head, or abstention gate. Keep cross-entropy for representation learning and add this objective during a calibration fine-tuning stage; treat the decision threshold \(t\) as an explicit post-training parameter rather than modifying hidden layers. (2) Pseudocode: split data into training, calibration, and test sets. Train \(f_\theta\) with cross-entropy. On each calibration batch or over the full calibration set, compute scores, choose a threshold \(t\), and count \(k_0=\sum_{x\in D_0}\mathbf 1[s_\theta(x)\geq t]\) and \(k_1=\sum_{x\in D_1}\mathbf 1[s_\theta(x)\geq t]\). Set \(\widetilde p_0=(k_0+1)/(n_0+2)\), \(\widetilde p_1=(k_1+1)/(n_1+2)\), then compute FPR, FNR, both KL terms, and the displayed loss. For differentiable training, replace the indicator by \(\sigma((s-t)/\tau)\), annealing \(\tau\); for final deployment, scan sorted validation scores. Select the smallest threshold whose upper-confidence FPR is at most \(\alpha\) and whose upper-confidence FNR is at most \(\beta\), using Clopper-Pearson or binomial-KL confidence bounds. (3) The paper supplies the separate-error and directed-divergence construction; probabilities, threshold, and confidence bounds are estimated empirically. (4) First experiment: train a small ResNet on CIFAR-10 versus an OOD split, or an MLP on a binary safety dataset. Compare cross-entropy with ordinary threshold tuning against this method at \((\alpha,\beta)=(10^{-2},10^{-2})\), \((10^{-3},10^{-2})\), and \((10^{-2},10^{-3})\). On a fresh test set measure worst-group FPR/FNR, constraint violations, AUROC, and calibration-set size. Success is fewer violations of both prescribed constraints at equal model size and accuracy, particularly when the targets are asymmetric, with negligible inference overhead.