✗ Failed on benchmark 2026

PAC transition-cover training monitor

Implementation & benchmark of arXiv:2608.10254 — A Pragmatic Guide to Building Conservative Discrete Abstractions of Cyber-Physical Systems

Usefulness7/10
Difficulty5/10
Novelty7/10

Source paper: A Pragmatic Guide to Building Conservative Discrete Abstractions of Cyber-Physical Systems arXiv:2608.10254 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use PAC-certified sampling to estimate whether a neural transition model has adequately covered the reachable successor set of each latent-state cell. Cells with insufficient coverage receive additional rollouts, larger uncertainty margins, or increased training weight. This prevents a model from appearing stable merely because rare but dynamically important transitions were never sampled.

Formulas

$$\Pr(S_i\subseteq B_i)\geq 1-\delta_i.$$
$$\Pr(\text{a region of mass at least }\varepsilon_i\text{ is missed})=(1-\varepsilon_i)^{n_i}\leq e^{-n_i\varepsilon_i}.$$
$$n_i\geq\frac{1}{\varepsilon_i}\log\frac{1}{\delta_i}\quad\Longrightarrow\quad \Pr(\text{miss})\leq\delta_i.$$
$$\mathcal{L}_{\mathrm{cover}}(\theta)=\sum_i\max_{y\in\widehat{S}_i}\mathrm{dist}(y,B_i)^2+\lambda\sum_i\mathbf{1}[\text{coverage test for }i\text{ fails}].$$

Mathematical statement

Partition the state-input domain into cells $C_i\times U_i$. Let $S_i=F_\theta(C_i,U_i)$ be the true successor set and let $B_i$ be an estimated outer successor region. If a region has probability mass at least $\varepsilon_i$ but is missed by all $n_i$ independent samples, its miss probability is $(1-\varepsilon_i)^{n_i}\leq e^{-n_i\varepsilon_i}$. Therefore $n_i\geq\varepsilon_i^{-1}\log(1/\delta_i)$ makes the miss probability at most $\delta_i$. The method is probabilistic rather than deterministic: its guarantee requires an appropriate sampling distribution and independence or a justified dependent-sampling correction.

Implementation notes

Attach the monitor to a learned dynamics model used for model-predictive control, world modeling, or long-horizon RNN prediction. Discretize the reachable latent-state and input space into cells, maintain a replay buffer indexed by cell, and allocate fresh rollouts according to coverage deficits. For each cell, construct an empirical successor envelope using coordinatewise quantiles, conformal radii, or interval propagation, reserving held-out transitions for evaluation. Given target miss probability epsilon_i and confidence failure delta_i, require n_i at least log(1/delta_i) divided by epsilon_i before declaring a cell covered. Pseudocode: identify the current cell; collect or simulate transitions; fit B_i; evaluate held-out successors; if the violation rate exceeds the allowed threshold, enlarge B_i, collect more samples, or increase the loss weight for that cell. The paper supplies the conservative transition and PAC coverage mechanism; the loss and sampling scheduler are neural-network adaptations. The first experiment should use a small MLP predicting the next state of a noisy pendulum or Lorenz system, comparing uniform data collection with cell-indexed PAC collection at equal transition budgets. The predicted signature is that held-out miss rates remain below epsilon_i except in approximately a delta_i fraction of cells, while under-covered cells exhibit larger long-horizon rollout error.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a PAC transition-cover monitor with deficit-first cell allocation, reproducible Bernoulli miss-probability verification, and an equal-budget MLP transition experiment. The math mechanism manifested: across 20 (epsilon,n) settings, empirical miss probabilities matched (1-epsilon)^n with maximum absolute error 0.00444, and all four PAC threshold tests were below delta=0.1. The neural comparison showed a modest improvement for PAC collection (test MSE 0.01174 vs 0.01194; cell-3 MSE 0.07965 vs 0.08897), but this is only a weak toy signal rather than a demonstrated general sample-efficiency win.

Agent confidence
8/10
Baseline
Uniform collection, 640 transitions, test MSE=0.0119438, cell-3 MSE=0.0889726, rare-branch MSE=0.6314883, rare samples=9
Idea
PAC deficit-first collection, 640 transitions, test MSE=0.0117403, cell-3 MSE=0.0796502, rare-branch MSE=0.5347307, rare samples=10; required n=24 per cell and all cells covered

Limitations: Only a scalar synthetic transition system and one fixed seed were tested. The prototype uses total sample count as the PAC readiness condition and does not implement a learned outer successor envelope, conformal calibration, held-out violation monitoring, dependent-sampling corrections, long-horizon rollout evaluation, multiple seeds, or a realistic MPC/world-model workload.

How to run: python3 pac_transition_cover.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.10254)

Significantly WORSE than the tuned baseline (+1497.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0226
Idea mean
0.3602
Effect (Δ)
0.3376 (+1497.1%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±347.2%
Mechanism
Not confirmed ✗
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

The final benchmark used the reduced fallback budget of 400 training samples, 200 test samples, and 3 epochs because the original CPU protocol exceeded 40 minutes and the shared CUDA runtime hit a low-level clock assertion. The implementation uses empirical input-cell coverage rather than a learned outer successor envelope or conformal calibration, and long-horizon rollout/MPC control metrics were not tested.

How to run: CUDA_VISIBLE_DEVICES='' OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 /home/maxwelhelp/main/bin/python3 stage2_pac_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Failed ✗

Agent confidence: 9/10

Implemented PAC transition-cover training on the structurally matched registered dynamics track with the shared rnn_small architecture. The tuned baseline achieved test MSE 0.02255185, while the best PAC configuration achieved 0.36016439; paired delta was +0.33761254 with permutation p=0.0081, yielding the required verdict idea worse (significant). The trained-model mechanism signature was not confirmed: held-out miss rate was 0.3017 versus epsilon 0.1.

Baseline
best lr=0.006, mean test MSE=0.02255184983368963, std=0.009104915744183411
Idea
best lr=0.006, mean test MSE=0.3601643927395344, std=0.08035137726894388
Idea wins
0
P value
0.0081
Paired delta
0.33761254290584475

Benchmark result

Significantly WORSE than the tuned baseline (+1497.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0226
Idea mean
0.3602
Effect (Δ)
0.3376 (+1497.1%; negative = idea better)
Wins
0 / 8 paired seeds
p-value
0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect
±347.2%
Mechanism
Not confirmed ✗
Practical effect
Hurts
Baseline tuning
swept over 3 configs
Limitations:

The final benchmark used the reduced fallback budget of 400 training samples, 200 test samples, and 3 epochs because the original CPU protocol exceeded 40 minutes and the shared CUDA runtime hit a low-level clock assertion. The implementation uses empirical input-cell coverage rather than a learned outer successor envelope or conformal calibration, and long-horizon rollout/MPC control metrics were not tested.

How to run: CUDA_VISIBLE_DEVICES='' OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 /home/maxwelhelp/main/bin/python3 stage2_pac_bench.py

Artifacts

Implementation overview ⬇ Download all as ZIP 6 files · code, reports and structured results