✗ Failed on benchmark 2026

Conformal CBVF Safety Shield

Implementation & benchmark of arXiv:2608.13819 — Control Barrier--Value Functions under Partial Observability: Safety Guarantees via Conformal Prediction

Usefulness8/10
Difficulty6/10
Novelty7/10

Source paper: Control Barrier--Value Functions under Partial Observability: Safety Guarantees via Conformal Prediction arXiv:2608.13819 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Wrap an observation-based neural policy with a real-time safety filter that accounts for uncertainty in its latent-state estimate. The policy proposes an action, while a quadratic program minimally modifies that action so a control-barrier/value function remains nonnegative for every state inside a conformally calibrated error set.

Formulas

$$q_{1-alpha}=r_{(ceil((N+1)(1-alpha)))} ,\qquad r_i=||x_i-E_phi(y_i)||,\qquad P(||x-E_phi(y)||<=q_{1-alpha}) approximately >=1-alpha.$$
$$V_hat(t,z)=min_{||e||<=q_{1-alpha}} V(t,z+e),\qquad V_hat(t,z)>=0 implies V(t,x)>=0 when ||x-z||<=q_{1-alpha}.$$
$$u_star=argmin_{u in U} (1/2)||u-pi_theta(y)||_R^2$$
$$subject\ to\ min_{||e||<=q, d in D} [partial_t V(t,z+e)+grad_x V(t,z+e)^T f(z+e,u,d)+gamma V(t,z+e)]>=0.$$

Mathematical statement

Let x in R^n be the true state, y the observation, z=E_phi(y) the neural state estimate, and e=x-z the estimation error. Given N exchangeable calibration samples, compute residual scores r_i=||x_i-E_phi(y_i)|| and choose the conformal order statistic q_(1-alpha). Then the error event ||e||<=q has probability approximately at least 1-alpha under exchangeability. Let V(t,x) be a differentiable control-barrier/value function, with V>=0 defining the safe viability region, and let the dynamics be dot{x}=f(x,u,d), where u is the action and d belongs to a bounded disturbance set D. The robust estimated-state certificate is the minimum of V over the uncertainty ball. A sufficient online condition is that the worst-case derivative of V over all errors in the ball and disturbances in D is at least -gamma V, where gamma>=0. The shield solves for the action closest to the neural policy action subject to this robust inequality.

Implementation notes

1. Integration point: train an observation-based policy pi_theta(y), using an RNN, transformer, or recurrent world model, but execute it through a safety layer. The estimator E_phi produces z. Fit or provide a differentiable certificate V(t,x), and use automatic differentiation to obtain its state gradient. 2. Pseudocode: collect N calibration tuples (y_i,x_i), compute r_i=norm(x_i-E_phi(y_i)), and set q to the conformal order statistic. At runtime compute z=E_phi(y), u0=pi_theta(y), generate error candidates e_j by projected optimization inside the ball norm(e)<=q, enumerate disturbance vertices d_k when D is a polytope, and solve a quadratic program minimizing norm(u-u0)_R subject to the barrier derivative inequality for each pair (e_j,d_k). Apply the resulting u and repeat. 3. Computed versus estimated: q is computed directly from held-out residuals; V and f are model-based or learned; the inner worst-case minimization is approximated by projected gradient ascent, random samples, or conservative Lipschitz bounds. 4. First experiment: use a two-dimensional point-mass obstacle-avoidance simulator with noisy position observations, an MLP or GRU policy, and compare an unshielded policy with a nominal barrier filter and the conformal shield. Sweep alpha in {0.01,0.05,0.1,0.2} and calibration size N. The prediction is that empirical estimator coverage is near or above 1-alpha, unsafe-episode frequency decreases as alpha decreases, and intervention magnitude increases with q. A sharp feasibility boundary should appear when q approaches the obstacle-clearance margin: beyond that point V_hat becomes negative and the QP becomes frequently infeasible or strongly corrective.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a self-contained conformal CBVF toy experiment with split-conformal calibration, an analytic robust barrier shield, parameter sweeps, and closed-loop comparison. The math predictions manifested: mean coverage was 0.9133, 0.9029, 0.9000, and 0.9003 for N=20, 50, 200, and 1000 against a 0.90 target; the observed certificate boundary was q=0.50 versus predicted q=0.50; and the observed intervention slope was 0.800000 versus predicted gamma=0.8. The safety rollout had zero unsafe episodes for nominal and conformal shields, so this toy does not demonstrate a safety-rate win, while conformal intervention increased from 0.7638 nominal to 0.8937--0.9161.

Agent confidence
8/10
Baseline
Nominal q=0: unsafe_rate=0.000, mean_intervention=0.76385, infeasible_certificate_fraction=0.05413
Idea
Conformal q values alpha=.20/.10/.05 = 0.22372/0.25625/0.29310; unsafe_rate=0.000 for all; mean_intervention=0.89369/0.90391/0.91606; infeasible_certificate_fraction=0.08373/0.08760/0.09502
Math predictions
Coverage target 0.90: observed means 0.9133, 0.9029, 0.9000, 0.9003 for N=20,50,200,1000; boundary predicted/observed=0.50/0.50; intervention slope predicted/observed=0.8/0.8

Limitations: This is a low-dimensional analytic toy, not an RNN/transformer policy or a general QP implementation. The disturbance and estimator errors are simple exchangeable distributions, continuous-time guarantees were not tested between sampling points, and the nominal shield was already perfectly safe, preventing a meaningful empirical safety improvement comparison.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.13819)

Ahead of the baseline (−3.8%) but not statistically significant (p=0.688, wins 5 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0028
Idea mean
0.0027
Effect (Δ)
-0.0001 (−3.8%; negative = idea better)
Wins
5 / 8 paired seeds
p-value
0.688 (permutation test, 20 000 shuffles)
Smallest detectable effect
±20.0%
Mechanism
Not confirmed ✗
Practical effect
Inconclusive
Baseline tuning
swept over 3 configs
Limitations:

The implementation is a training-time robust-loss surrogate for the proposed real-time QP safety shield; it does not execute an online QP or measure closed-loop unsafe-episode rates. The benchmark uses flattened pendulum histories and 400 training plus 200 test samples per seed.

How to run: PYTHONPATH=/home/maxwelhelp/all/math2nn /home/maxwelhelp/main/bin/python3 stage2_conformal_cbvf.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 9/10

Implemented and ran the conformal CBVF-inspired robust training intervention on the registered dynamics track with the shared rnn_small architecture. The idea slightly reduced mean test MSE, but the paired permutation test was not significant and the trained-model mechanism signature was not confirmed; therefore the benchmark verdict is no significant win.

Baseline
Mean test MSE 0.0027758953, best lr=0.01 and 10 epochs.
Comparison
delta_mean=-0.0001043133, 5/8 wins, permutation p=0.688, verdict=no significant win.
Idea
Mean test MSE 0.0026715820, best lr=0.01, alpha=0.20, and 10 epochs.

Benchmark result

Ahead of the baseline (−3.8%) but not statistically significant (p=0.688, wins 5 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0028
Idea mean
0.0027
Effect (Δ)
-0.0001 (−3.8%; negative = idea better)
Wins
5 / 8 paired seeds
p-value
0.688 (permutation test, 20 000 shuffles)
Smallest detectable effect
±20.0%
Mechanism
Not confirmed ✗
Practical effect
Inconclusive
Baseline tuning
swept over 3 configs
Limitations:

The implementation is a training-time robust-loss surrogate for the proposed real-time QP safety shield; it does not execute an online QP or measure closed-loop unsafe-episode rates. The benchmark uses flattened pendulum histories and 400 training plus 200 test samples per seed.

How to run: PYTHONPATH=/home/maxwelhelp/all/math2nn /home/maxwelhelp/main/bin/python3 stage2_conformal_cbvf.py

Artifacts

Implementation overview ⬇ Download all as ZIP 6 files · code, reports and structured results