# Audited Risk-Budgeted Early Exit

- ID: 844
- Canonical URL: https://synthcore.org/idea/844/audited-risk-budgeted-early-exit
- API JSON: https://synthcore.org/api/idea/844.json
- API Markdown: https://synthcore.org/api/idea/844.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.13221](https://arxiv.org/abs/2607.13221)
- Category: other
- Solves: speedup, stability, generalization
- ML areas: inference-speedup, transformer, mlp, regularization
- Math tags: probability, statistics, optimization
- Ratings: usefulness 8/10; difficulty 4/10; novelty 7/10

## Idea description

Attach a cheap risk score to each neural-network prediction and skip an expensive verifier, ensemble, diffusion refinement, retrieval call, or human review when the score is below a calibrated threshold. Independently audit a random subset of skipped examples using the expensive ground-truth procedure, and select the largest skip threshold whose exact confidence bound keeps the violation rate below a target budget.

## Mathematical statement

For a deployment window with finite item set \(\mathcal{C}\), each item \(i\) has cheap score \(r_i\), binary violation label \(V_i\in\{0,1\}\) revealed only by the expensive verifier, and candidate skip set \(S(\tau)=\{i:r_i\leq\tau\}\). Draw audit indicators \(B_i\sim\operatorname{Bernoulli}(\pi)\) independently of all scores and labels, so \(A=\{i:B_i=1\}\). For threshold \(\tau\), let \(n_A(\tau)=\sum_{i\in A}\mathbf{1}[i\in S(\tau)]\) and \(k_A(\tau)=\sum_{i\in A}\mathbf{1}[i\in S(\tau)]V_i\). The Clopper-Pearson upper confidence limit is \(U_\delta(k,n)=\operatorname{Beta}^{-1}(1-\delta;k+1,n-k)\), with \(U_\delta(0,0)=0\) by convention. Accept the first data-independent candidate threshold for which \(U_{\delta_j}(k_A(\tau),n_A(\tau))\leq\alpha\), where \(\alpha\) is the allowed violation rate. If testing \(M\) thresholds, use \(\delta_j=\delta/M\) and a union bound. The predicted finite-window signature is that an accepted skipped set has true violation rate at most \(\alpha\) with confidence approximately \(1-\delta\), while compute cost decreases monotonically as \(\tau\) increases.

## Key formulas

- $$B_i\sim\operatorname{Bernoulli}(\pi),\qquad A=\{i\in\mathcal{C}:B_i=1\},\qquad B_i\perp\!\perp\{r_j,V_j\}_{j\in\mathcal{C}}.$$
- $$S(\tau)=\{i\in\mathcal{C}:r_i\leq\tau\},\quad n_A(\tau)=\sum_{i\in A}\mathbf{1}\{i\in S(\tau)\},\quad k_A(\tau)=\sum_{i\in A}\mathbf{1}\{i\in S(\tau)\}V_i.$$
- $$U_\delta(k,n)=\operatorname{Beta}^{-1}(1-\delta;k+1,n-k),\qquad \text{accept }\tau\ \Longleftrightarrow\ U_{\delta_j}(k_A(\tau),n_A(\tau))\leq\alpha.$$
- $$\Pr\left(\operatorname{ViolRate}(S(\hat\tau))\leq\alpha\right)\geq 1-\delta,\qquad \operatorname{ComputeCost}(\tau)\approx C_{\mathrm{cheap}}+C_{\mathrm{exp}}\left(1-\frac{|S(\tau)|}{|\mathcal{C}|}\right).$$

## Implementation notes

1. Integration point: place this layer after a cheap neural prediction and before an expensive operation. For a first implementation, use a ResNet-18 or small ViT on CIFAR-10/CIFAR-10-C; let the cheap score be negative maximum softmax probability or a learned risk head, and let the expensive verifier be a five-model ensemble or a larger calibrated model whose disagreement/error indicator is treated as \(V_i\). In production analogues, the verifier could be a robust optimization solve, retrieval call, diffusion refinement, or human review. 2. Pseudocode: collect a deployment window \(\mathcal{C}\); compute \(r_i\) for every item; independently sample \(B_i\) with probability \(\pi\); run the expensive verifier only when \(B_i=1\); sort candidate thresholds from strict to permissive; for each threshold compute \(n\) and \(k\) among audited skipped items; calculate CP upper bound \(\operatorname{BetaInverse}(1-\delta_j,k+1,n-k)\); choose the first threshold with \(U\leq\alpha\); skip the expensive operation for all subsequent items with \(r_i\leq\tau\). If no threshold passes, verify everything. 3. Computed quantities are audit membership, verifier labels, \(k\), \(n\), and the exact beta inverse; the score and its relation to risk are estimated by the neural model, but are not assumed correct. The audit must be sampled independently after scores are produced, and threshold candidates must be fixed before observing audit labels or receive a multiple-testing correction. 4. Cheap experiment: compare always-verifying, a fixed confidence threshold, and audited selection on CIFAR-10-C with shifted corruption severity. Use \(\alpha=0.05\), \(\delta=0.05\), audit rate \(\pi=0.1\), and 20 candidate thresholds with \(\delta_j=0.05/20\). Measure verifier calls and violations on the non-audited skipped set. The falsifiable prediction is that accepted windows have skipped-set violation rate below 5% in at least 95% of repeated windows, whereas fixed-threshold screening can exceed 5% under shift. Increasing \(\tau\) should reduce compute monotonically until the confidence bound crosses \(\alpha\).

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built an auditable early-exit selector using exact Clopper–Pearson upper bounds with Bonferroni correction across 20 fixed thresholds, plus a seeded finite-window toy experiment. The mechanism manifested: CP coverage was at least the predicted 95% in every binomial sweep (observed 95.36%–100%), simultaneous threshold failure was 0.1% versus the 5% target, and compute cost decreased monotonically with threshold. Audited selection achieved 0% of shifted windows over the 5% violation budget while skipping 16.6% on average; a fixed source-calibrated threshold skipped more but had 0.25% shifted-window violations above budget. No neural-network or real CIFAR experiment was run, so the result validates the statistical mechanism rather than end-to-end deployment speedup.

### Mechanism check

- Verdict: Built an auditable early-exit selector using exact Clopper–Pearson upper bounds with Bonferroni correction across 20 fixed thresholds, plus a seeded finite-window toy experiment. The mechanism manifested: CP coverage was at least the predicted 95% in every binomial sweep (observed 95.36%–100%), simultaneous threshold failure was 0.1% versus the 5% target, and compute cost decreased monotonically with threshold. Audited selection achieved 0% of shifted windows over the 5% violation budget while skipping 16.6% on average; a fixed source-calibrated threshold skipped more but had 0.25% shifted-window violations above budget. No neural-network or real CIFAR experiment was run, so the result validates the statistical mechanism rather than end-to-end deployment speedup.
- Confidence: 8/10
- Limitations: Only a synthetic score/violation process was tested. Verifier cost was modeled abstractly, and no neural model, CIFAR/CIFAR-C shift, calibration training, adaptive threshold candidate generation, audit latency, or production integration was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/8
- Benchmark verdict: no measurable effect

## Artifacts

- [audited_exit.py](https://synthcore.org/code/155/audited_exit.py)
- [audited_exit_bench.py](https://synthcore.org/code/155/audited_exit_bench.py)
- [bench_report.json](https://synthcore.org/code/155/bench_report.json)
- [report.md](https://synthcore.org/code/155/report.md)
- [report_bench_2026-08-30T180029.md](https://synthcore.org/code/155/report_bench_2026-08-30T180029.md)
- [results.json](https://synthcore.org/code/155/results.json)
- [Download all files as ZIP](https://synthcore.org/download/155)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.