✗ Failed on benchmark 2026

Projective Boundary Certificates for Neural Selective Prediction

Implementation & benchmark of arXiv:2609.01355 — Exact Risk-Complexity Laws for Projective Boundaries in Scenario Optimization and Distribution-Free Certification

Usefulness8/10

Difficulty5/10

Novelty7/10

Theory Generalization Stability Mlp Transformer Regularization Probability Statistics Combinatorics Optimization Learning theory

Source paper: Exact Risk-Complexity Laws for Projective Boundaries in Scenario Optimization and Distribution-Free Certification arXiv:2609.01355 ⓘ · analyzed Sep 2, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Construct a neural acceptance or abstention set from calibration samples together with an explicit boundary map selecting the samples that determine the set. If the map is proper projective and its cross-sample complexity profile is stable, the conditional violation risk has an exact beta law indexed by boundary size rather than network parameter count. This provides a falsifiable, distribution-free certificate for neural selective classifiers and learned safety filters.

Formulas

$$R(S)=\mathbb{P}_{Z\sim P}\left\{Z\notin A(S)\mid S\right\},\qquad K(S)=\left|B(S)\right|.$$

$$z_i\in A(S\setminus\{z_i\})\quad\Longleftrightarrow\quad B(S\setminus\{z_i\})=B(S).$$

$$x\in A(S),\ x\notin B(S)\quad\Longrightarrow\quad B(S\setminus\{x\})=B(S).$$

$$R\mid K=k\sim\operatorname{Beta}(k,n+1-k)\quad\text{if the cross-sample profile is stable};\qquad f_{R\mid K=k}(r)=\sum_c w(c\mid k)f_{R\mid c}(r)\quad\text{otherwise}.$$

Mathematical statement

Let S=(z_1,...,z_n) be an exchangeable calibration sample, A(S) the acceptance set for a fresh point z, and B(S) a boundary map. Define R(S)=P(z is not in A(S) given S) and K(S)=|B(S)|. Proper projectivity requires that a held-out point z_i is accepted by the leave-one-out rule exactly when the full-sample boundary is retained, and that deleting any accepted non-boundary point does not change B(S). If the cross-sample complexity profile q_k is stable, exchangeability gives a beta law, with indexing determined by the boundary convention; a common convention is R given K=k distributed as Beta(k,n+1-k). If q_k varies, conditioning only on K is invalid and the conditional law must be represented as a mixture over boundary configurations c with weights w(c given k).

Implementation notes

1. Integration point: train a neural predictor on a training split, then construct an acceptance rule on an independent calibration split. For selective classification, let s_theta(x,y) be a nonconformity score and accept when s_theta(x,y) is at most a calibrated threshold t(S). For a safety filter, use several scores and a vector threshold. Store an explicit boundary map consisting of points attaining the active order statistic or active Pareto/support constraints. 2. Pseudocode: compute calibration scores; construct A(S) and B(S); record K; for each calibration index i, delete z_i and recompute the rule; test whether deleting i preserves the boundary and whether z_i is accepted exactly when the boundary is preserved. Repeat over random subsets to estimate the cross-sample profile. If the profile is stable, calculate the beta upper quantile BetaInv(1-delta;k,n+1-k). If it is unstable, estimate mixture weights over boundary configurations and use the corresponding mixture quantile. 3. Taken from the paper: projectivity, deletion invariance, boundary-size conditioning, and the beta-versus-mixture distinction. Estimated empirically: profile stability, deletion invariance, and mixture weights. 4. First experiment: use an MLP on MNIST with an independent calibration split and top-score abstention. Compare ordinary conformal calibration, the beta certificate, and a naive boundary-size certificate. Repeat over at least 1,000 test batches. The prediction is that scalar order-statistic rules will match the beta CDF, while tied or competing vector boundaries will deviate from it but match the estimated profile mixture; certified violation frequency should remain below delta.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a compact projective-boundary MVP for scalar max-score calibration plus an unstable adaptive-rank control. The max rule matched the predicted Beta(1,20) moments closely, passed projectivity checks, and achieved approximately 0.9 certificate coverage; the unstable control's naive boundary-size beta certificate covered only 0.404 of batches. The idea provides a real exact-law diagnostic/certificate, but it did not improve predictive coverage over ordinary conformal calibration because both use the same max threshold.

Agent confidence: 8/10

Baseline: Ordinary split-conformal max threshold: certificate coverage 0.889 over 1,000 batches; mean true risk 0.04745.
Idea: Projective-boundary Beta(1,20) certificate: coverage 0.889, with empirical moments matching theory; scalar deletion equivalence 0.95 and projectivity 1.0. Unstable rank control using the same naive beta law: coverage 0.404 and mean risk 0.2284.

Limitations: The MVP uses synthetic uniform nonconformity scores rather than training an actual MLP or testing MNIST. It verifies the scalar order-statistic theorem and demonstrates failure under an unstable boundary profile, but does not evaluate vector/Pareto boundaries, estimated profile mixtures, computational speed, or neural selective-prediction accuracy.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.01355) Sep 2, 2026

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 20.8817
Idea mean: 20.8817
Effect (Δ): 0 (−-0.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the registered tabular track was tested; vision, sequence, and dynamics were not structurally necessary for scalar calibration residual certificates. Vector/Pareto boundaries, mixture-profile certificates, and full multiclass selective classification were not tested. The intervention did not change training, so no task-MSE improvement was expected.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 10/10

The full stage-2 bench was run on the registered tabular track with eight paired seeds and a tuned three-configuration baseline sweep. The idea and baseline are identical trained systems because the certificate is an inference-time diagnostic, yielding exactly equal test MSE (delta_mean=0, p=1.0); the trained residual boundary also failed the predicted projective mechanism (mean K=15, projectivity=0.85), so this is not a win.

Baseline: mlp_tiny tabular regression; best lr=0.006, epochs=12, alpha=0.15; full 8-seed MSE mean 20.881689, std 2.809771.
Idea: Same paired neural systems with explicit residual boundary certificate; full 8-seed MSE mean 20.881689, std 2.809771; delta_mean 0.0, p=1.0.

Benchmark result