✗ Failed on benchmark 2026

Audited Risk-Budgeted Early Exit

Implementation & benchmark of arXiv:2607.13221 — Audited Selective Verification for Risk-Controlled N-1 Thermal Contingency Screening under Deployment Shift

Usefulness8/10

Difficulty4/10

Novelty7/10

Other Speedup Stability Generalization Inference speedup Transformer Mlp Regularization Probability Statistics Optimization

Source paper: Audited Selective Verification for Risk-Controlled N-1 Thermal Contingency Screening under Deployment Shift arXiv:2607.13221 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Attach a cheap risk score to each neural-network prediction and skip an expensive verifier, ensemble, diffusion refinement, retrieval call, or human review when the score is below a calibrated threshold. Independently audit a random subset of skipped examples using the expensive ground-truth procedure, and select the largest skip threshold whose exact confidence bound keeps the violation rate below a target budget.

Formulas

$$B_i\sim\operatorname{Bernoulli}(\pi),\qquad A=\{i\in\mathcal{C}:B_i=1\},\qquad B_i\perp\!\perp\{r_j,V_j\}_{j\in\mathcal{C}}.$$

$$S(\tau)=\{i\in\mathcal{C}:r_i\leq\tau\},\quad n_A(\tau)=\sum_{i\in A}\mathbf{1}\{i\in S(\tau)\},\quad k_A(\tau)=\sum_{i\in A}\mathbf{1}\{i\in S(\tau)\}V_i.$$

$$U_\delta(k,n)=\operatorname{Beta}^{-1}(1-\delta;k+1,n-k),\qquad \text{accept }\tau\ \Longleftrightarrow\ U_{\delta_j}(k_A(\tau),n_A(\tau))\leq\alpha.$$

$$\Pr\left(\operatorname{ViolRate}(S(\hat\tau))\leq\alpha\right)\geq 1-\delta,\qquad \operatorname{ComputeCost}(\tau)\approx C_{\mathrm{cheap}}+C_{\mathrm{exp}}\left(1-\frac{|S(\tau)|}{|\mathcal{C}|}\right).$$

Mathematical statement

For a deployment window with finite item set $\mathcal{C}$, each item $i$ has cheap score $r_i$, binary violation label $V_i\in\{0,1\}$ revealed only by the expensive verifier, and candidate skip set $S(\tau)=\{i:r_i\leq\tau\}$. Draw audit indicators $B_i\sim\operatorname{Bernoulli}(\pi)$ independently of all scores and labels, so $A=\{i:B_i=1\}$. For threshold $\tau$, let $n_A(\tau)=\sum_{i\in A}\mathbf{1}[i\in S(\tau)]$ and $k_A(\tau)=\sum_{i\in A}\mathbf{1}[i\in S(\tau)]V_i$. The Clopper-Pearson upper confidence limit is $U_\delta(k,n)=\operatorname{Beta}^{-1}(1-\delta;k+1,n-k)$, with $U_\delta(0,0)=0$ by convention. Accept the first data-independent candidate threshold for which $U_{\delta_j}(k_A(\tau),n_A(\tau))\leq\alpha$, where $\alpha$ is the allowed violation rate. If testing $M$ thresholds, use $\delta_j=\delta/M$ and a union bound. The predicted finite-window signature is that an accepted skipped set has true violation rate at most $\alpha$ with confidence approximately $1-\delta$, while compute cost decreases monotonically as $\tau$ increases.

Implementation notes

1. Integration point: place this layer after a cheap neural prediction and before an expensive operation. For a first implementation, use a ResNet-18 or small ViT on CIFAR-10/CIFAR-10-C; let the cheap score be negative maximum softmax probability or a learned risk head, and let the expensive verifier be a five-model ensemble or a larger calibrated model whose disagreement/error indicator is treated as $V_i$. In production analogues, the verifier could be a robust optimization solve, retrieval call, diffusion refinement, or human review. 2. Pseudocode: collect a deployment window $\mathcal{C}$; compute $r_i$ for every item; independently sample $B_i$ with probability $\pi$; run the expensive verifier only when $B_i=1$; sort candidate thresholds from strict to permissive; for each threshold compute $n$ and $k$ among audited skipped items; calculate CP upper bound $\operatorname{BetaInverse}(1-\delta_j,k+1,n-k)$; choose the first threshold with $U\leq\alpha$; skip the expensive operation for all subsequent items with $r_i\leq\tau$. If no threshold passes, verify everything. 3. Computed quantities are audit membership, verifier labels, $k$, $n$, and the exact beta inverse; the score and its relation to risk are estimated by the neural model, but are not assumed correct. The audit must be sampled independently after scores are produced, and threshold candidates must be fixed before observing audit labels or receive a multiple-testing correction. 4. Cheap experiment: compare always-verifying, a fixed confidence threshold, and audited selection on CIFAR-10-C with shifted corruption severity. Use $\alpha=0.05$, $\delta=0.05$, audit rate $\pi=0.1$, and 20 candidate thresholds with $\delta_j=0.05/20$. Measure verifier calls and violations on the non-audited skipped set. The falsifiable prediction is that accepted windows have skipped-set violation rate below 5% in at least 95% of repeated windows, whereas fixed-threshold screening can exceed 5% under shift. Increasing $\tau$ should reduce compute monotonically until the confidence bound crosses $\alpha$.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built an auditable early-exit selector using exact Clopper–Pearson upper bounds with Bonferroni correction across 20 fixed thresholds, plus a seeded finite-window toy experiment. The mechanism manifested: CP coverage was at least the predicted 95% in every binomial sweep (observed 95.36%–100%), simultaneous threshold failure was 0.1% versus the 5% target, and compute cost decreased monotonically with threshold. Audited selection achieved 0% of shifted windows over the 5% violation budget while skipping 16.6% on average; a fixed source-calibrated threshold skipped more but had 0.25% shifted-window violations above budget. No neural-network or real CIFAR experiment was run, so the result validates the statistical mechanism rather than end-to-end deployment speedup.

Agent confidence: 8/10

Baseline: Fixed threshold calibrated on unshifted synthetic data: threshold 0.95, shifted mean skipped-set risk 0.0360, 0.25% of windows over alpha; always-verify skipped fraction 0.
Idea: Audited CP selector: shifted mean skipped-set risk 0.00375, 0% of 400 windows over alpha, mean skipped fraction 0.1657; simultaneous CP failure 0.001 over 3000 trials versus 0.05 target; monotone compute-cost check passed. Direct CP coverage sweep ranged from 0.9536 to 1.0 versus the 0.95 target.

Limitations: Only a synthetic score/violation process was tested. Verifier cost was modeled abstractly, and no neural model, CIFAR/CIFAR-C shift, calibration training, adaptive threshold candidate generation, audit latency, or production integration was evaluated.

How to run: python3 audited_exit.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.13221) Sep 2, 2026

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark: Vision (image classification)
Model: cnn_small
Paired seeds: 8
Baseline mean: 0.5447
Idea mean: 0.5447
Effect (Δ): 0 (−-0.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the registered built-in vision track was tested. No CIFAR-10-C shift, larger verifier, direct latency measurement, or production audit scheduling was evaluated.

How to run: python3 audited_exit_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 10/10

The audited CP early-exit system was evaluated on the registered vision/cnn_small track with 8 paired seeds, a tuned baseline sweep, shared learning-rate search, and the bench permutation test. It matched the baseline on standard test error (delta 0.0, p=1.0), skipped only 1.01% on average, and did not confirm the risk-budget mechanism because the maximum observed skipped risk was 0.2174, above alpha=0.05.

Baseline: vision/cnn_small, best lr=0.003, epochs=6, mean test error 0.5446672179890817, std 0.020104511615005757
Idea: audited CP gate, best lr=0.003, epochs=6, mean test error 0.5446672179890817, std 0.020104511615005757, mean skip fraction 0.01009169041059136, mean cost ratio 0.9899083095894087, maximum skip risk 0.21739130434782608

Benchmark result