SynthCore
△ Mechanism confirmed, baseline not beaten 2026

Conformal Lower-Clearance Certificate for Neural Selectors

Implementation & benchmark of arXiv:2608.26533 — Barrier Function Conformal Safety Clearance Certification with CVaR for Driving Trajectory Selection

Usefulness8/10
Difficulty4/10
Novelty5/10
Theory StabilityGeneralizationAccuracyOptimizerRlWorld modelTraining dynamicsProbabilityStatisticsControl theory

Source paper: Barrier Function Conformal Safety Clearance Certification with CVaR for Driving Trajectory Selection arXiv:2608.26533 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Attach a finite-sample lower safety certificate to the trajectory selected by a neural planner or policy by calibrating the difference between predicted and realized clearance. A lower-tail CVaR of sampled neural predictions can provide the raw margin, while conformal calibration subtracts an empirical correction that absorbs predictor bias and sampling error.

Formulas

$$R_i=M_i-Y_i,\qquad q=\operatorname{Quantile}_{1-\alpha}\{R_1,\ldots,R_n\},\qquad C(I)=M(I)-q.$$
$$\Pr\left(Y(I_{n+1})\ge M(I_{n+1})-q\right)\ge 1-\alpha\quad\text{under exchangeability}.$$
$$M_{\mathrm{CVaR},\gamma}(I)=\max_{\eta\in\mathbb{R}}\left[\eta-\frac{1}{\gamma K}\sum_{k=1}^{K}(\eta-Z_k(I))_+\right].$$
$$q=R_{(k)},\qquad k=\left\lceil(n+1)(1-\alpha)\right\rceil,\qquad R_{(1)}\le\cdots\le R_{(n)}.$$

Mathematical statement

Let I_i be a planning episode, Y_i the realized minimum signed clearance of the trajectory selected using I_i, and M_i its predicted safety margin. Define the nonconformity score R_i=M_i-Y_i. For miscoverage alpha and n calibration episodes, choose q as the suitably rounded empirical (1-alpha)-quantile of R_i. The deployed certificate is C(I)=M(I)-q. Under exchangeability of calibration and test episodes, the probability that realized clearance is at least the certificate is at least 1-alpha. If Z_1,...,Z_K are sampled predicted clearances and gamma is the lower-tail probability, use the empirical CVaR M_CVaR=max_eta[eta-(gamma K)^(-1) sum_k(eta-Z_k)_+] as M before calibration.

Implementation notes

1. Integration point: leave the neural planner and candidate-selection rule unchanged, but make the network output either a nominal minimum-clearance prediction or K stochastic clearance predictions. For each episode, compute M for the trajectory actually selected by the planner and record its realized minimum signed OBB clearance after rollout. 2. Pseudocode: collect n calibration sessions; compute M_i and Y_i; set R_i=M_i-Y_i; sort the residuals; select q=R_(ceil((n+1)(1-alpha))); during deployment return C=M_new-q. Reject, replan, or reduce speed when C<0. If using samples, first optimize the one-dimensional CVaR expression over eta. 3. Computed from the mechanism: residual definition, order-statistic quantile, and CVaR formula. Estimated empirically: residual distribution, prediction samples, and validity of exchangeability. Selection must occur before R_i is computed, otherwise the certificate does not cover the selected output. 4. First experiment: use a frozen nuPlan planner or a small neural trajectory scorer on 300 logged sessions, with alpha=0.1, comparing nominal and CVaR margins. The predicted signature is held-out coverage P(Y>=C)>=0.9, typically near 0.90 to 0.97, while increasing alpha to 0.2 should reduce q and move coverage toward 0.8. Report q, certificate nonnegativity, and coverage confidence intervals. Deliberately shift test scenes to expose the expected failure under distribution shift.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a synthetic conformal lower-clearance certificate with empirical lower-tail CVaR and finite-sample order-statistic calibration. The CVaR optimizer matched the direct lower-tail computation to 2.2e-16; under exchangeability, coverage tracked the predicted targets (alpha=.05/.1/.2/.3: .950/.901/.804/.702), while uncalibrated CVaR coverage at alpha=.1 was only .771. Smaller gamma monotonically reduced the calibrated residual correction (q=-.009, .025, .072, .164 for gamma=.1,.25,.5,1), and the deliberate test shift reduced coverage to .734, demonstrating the expected exchangeability limitation.

Agent confidence
9/10
Baseline
Uncalibrated CVaR certificate C=M: coverage 0.771 at alpha=0.1; under shifted test distribution 0.554.
Idea
Calibrated CVaR: coverage 0.901 at alpha=0.1 with q=0.0249 and certificate nonnegativity 0.9997; shifted coverage 0.734. CVaR identity max error 2.2e-16. Alpha sweep coverage: 0.950, 0.901, 0.804, 0.702 for alpha 0.05, 0.1, 0.2, 0.3.

Limitations: This is a synthetic selected-trajectory simulator, not a nuPlan or real neural-planner integration; no learned model, OBB geometry, planner replanning, confidence intervals, or GPU benchmark was tested. Distribution-shift failure means the formal guarantee should not be applied outside the calibration distribution.

How to run: python3 certificate_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.26533)

Significantly WORSE than the tuned baseline (+91.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0059
Idea mean
0.0112
Effect (Δ)
0.0054 (+91.1%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±60.6%
Mechanism
Confirmed ✓
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

The built-in dynamics task predicts terminal pendulum angle rather than an explicit planner-selected multi-step trajectory or OBB clearance, so clearance was operationalized as 1.5 minus absolute terminal angle. No real planner, collision geometry, distribution-shift stress test, or larger model was tested. The canonical training loop emitted a broadcasting warning because the K-head output was trained against a scalar target; both paired systems used the same setup, but this should be corrected in a follow-up experiment.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 9/10

Implemented the conformal lower-clearance certificate on the matched dynamics RNN benchmark with paired seeds, equal learning-rate sweeps, empirical lower-tail CVaR, and conformal calibration. The trained-model mechanism signature was confirmed: mean clearance coverage was 0.90094 for the 0.90 target and the CVaR identity error was 1.11e-16. The standard task metric worsened significantly, however, with idea MSE 0.0112481 versus baseline 0.0058871, paired delta +0.0053610 and permutation p=0.0081; therefore the idea did not win.

Baseline
dynamics/rnn_small MSE 0.0058870674, best lr 0.006; sweep means: lr 0.001=0.01659585, lr 0.003=0.01240547, lr 0.006=0.00576727
Idea
dynamics/rnn_small MSE 0.0112481067, best lr 0.006; paired delta +0.0053610393, permutation p=0.0081; trained-model coverage 0.9009375, mean q 0.0517707, certificate nonnegative fraction 0.92, CVaR max error 1.11e-16

Benchmark result

Significantly WORSE than the tuned baseline (+91.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0059
Idea mean
0.0112
Effect (Δ)
0.0054 (+91.1%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±60.6%
Mechanism
Confirmed ✓
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

The built-in dynamics task predicts terminal pendulum angle rather than an explicit planner-selected multi-step trajectory or OBB clearance, so clearance was operationalized as 1.5 minus absolute terminal angle. No real planner, collision geometry, distribution-shift stress test, or larger model was tested. The canonical training loop emitted a broadcasting warning because the K-head output was trained against a scalar target; both paired systems used the same setup, but this should be corrected in a follow-up experiment.

How to run: python3 bench_experiment.py

Artifacts

Implementation overview ⬇ Download all as ZIP 6 files · code, reports and structured results

Related ideas