PAC transition-cover training monitor
Implementation & benchmark of arXiv:2608.10254 — A Pragmatic Guide to Building Conservative Discrete Abstractions of Cyber-Physical Systems
Source paper: A Pragmatic Guide to Building Conservative Discrete Abstractions of Cyber-Physical Systems arXiv:2608.10254 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use PAC-certified sampling to estimate whether a neural transition model has adequately covered the reachable successor set of each latent-state cell. Cells with insufficient coverage receive additional rollouts, larger uncertainty margins, or increased training weight. This prevents a model from appearing stable merely because rare but dynamically important transitions were never sampled.
Formulas
Mathematical statement
Partition the state-input domain into cells $C_i\times U_i$. Let $S_i=F_\theta(C_i,U_i)$ be the true successor set and let $B_i$ be an estimated outer successor region. If a region has probability mass at least $\varepsilon_i$ but is missed by all $n_i$ independent samples, its miss probability is $(1-\varepsilon_i)^{n_i}\leq e^{-n_i\varepsilon_i}$. Therefore $n_i\geq\varepsilon_i^{-1}\log(1/\delta_i)$ makes the miss probability at most $\delta_i$. The method is probabilistic rather than deterministic: its guarantee requires an appropriate sampling distribution and independence or a justified dependent-sampling correction.
Implementation notes
Attach the monitor to a learned dynamics model used for model-predictive control, world modeling, or long-horizon RNN prediction. Discretize the reachable latent-state and input space into cells, maintain a replay buffer indexed by cell, and allocate fresh rollouts according to coverage deficits. For each cell, construct an empirical successor envelope using coordinatewise quantiles, conformal radii, or interval propagation, reserving held-out transitions for evaluation. Given target miss probability epsilon_i and confidence failure delta_i, require n_i at least log(1/delta_i) divided by epsilon_i before declaring a cell covered. Pseudocode: identify the current cell; collect or simulate transitions; fit B_i; evaluate held-out successors; if the violation rate exceeds the allowed threshold, enlarge B_i, collect more samples, or increase the loss weight for that cell. The paper supplies the conservative transition and PAC coverage mechanism; the loss and sampling scheduler are neural-network adaptations. The first experiment should use a small MLP predicting the next state of a noisy pendulum or Lorenz system, comparing uniform data collection with cell-indexed PAC collection at equal transition budgets. The predicted signature is that held-out miss rates remain below epsilon_i except in approximately a delta_i fraction of cells, while under-covered cells exhibit larger long-horizon rollout error.
Verification
Stage 1 · Toy mechanism gate: Passed ✓
Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗
Stage 2 · Practical benchmark result: Significantly worse ✗
Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works
Stage 1 — Mechanism check agent confidence 8/10
Built a PAC transition-cover monitor with deficit-first cell allocation, reproducible Bernoulli miss-probability verification, and an equal-budget MLP transition experiment. The math mechanism manifested: across 20 (epsilon,n) settings, empirical miss probabilities matched (1-epsilon)^n with maximum absolute error 0.00444, and all four PAC threshold tests were below delta=0.1. The neural comparison showed a modest improvement for PAC collection (test MSE 0.01174 vs 0.01194; cell-3 MSE 0.07965 vs 0.08897), but this is only a weak toy signal rather than a demonstrated general sample-efficiency win.
- Agent confidence
- 8/10
- Baseline
- Uniform collection, 640 transitions, test MSE=0.0119438, cell-3 MSE=0.0889726, rare-branch MSE=0.6314883, rare samples=9
- Idea
- PAC deficit-first collection, 640 transitions, test MSE=0.0117403, cell-3 MSE=0.0796502, rare-branch MSE=0.5347307, rare samples=10; required n=24 per cell and all cells covered
Limitations: Only a scalar synthetic transition system and one fixed seed were tested. The prototype uses total sample count as the PAC readiness condition and does not implement a learned outer successor envelope, conformal calibration, held-out violation monitoring, dependent-sampling corrections, long-horizon rollout evaluation, multiple seeds, or a realistic MPC/world-model workload.
How to run: python3 pac_transition_cover.py
Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.10254)
Significantly WORSE than the tuned baseline (+1497.1%, p=0.0081, wins 0 / 8 paired seeds).
- Benchmark
- Dynamics forecasting
- Model
- rnn_small
- Paired seeds
- 8
- Baseline mean
- 0.0226
- Idea mean
- 0.3602
- Effect (Δ)
- 0.3376 (+1497.1%; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 0.0081 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- ±347.2%
- Mechanism
- Not confirmed ✗
- Practical effect
- Hurts
- Baseline tuning
- swept over 3 configs
The final benchmark used the reduced fallback budget of 400 training samples, 200 test samples, and 3 epochs because the original CPU protocol exceeded 40 minutes and the shared CUDA runtime hit a low-level clock assertion. The implementation uses empirical input-cell coverage rather than a learned outer successor envelope or conformal calibration, and long-horizon rollout/MPC control metrics were not tested.
How to run: CUDA_VISIBLE_DEVICES='' OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 /home/maxwelhelp/main/bin/python3 stage2_pac_bench.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Stage 2 — Benchmark latest Failed ✗
Agent confidence: 9/10
Implemented PAC transition-cover training on the structurally matched registered dynamics track with the shared rnn_small architecture. The tuned baseline achieved test MSE 0.02255185, while the best PAC configuration achieved 0.36016439; paired delta was +0.33761254 with permutation p=0.0081, yielding the required verdict idea worse (significant). The trained-model mechanism signature was not confirmed: held-out miss rate was 0.3017 versus epsilon 0.1.
- Baseline
- best lr=0.006, mean test MSE=0.02255184983368963, std=0.009104915744183411
- Idea
- best lr=0.006, mean test MSE=0.3601643927395344, std=0.08035137726894388
- Idea wins
- 0
- P value
- 0.0081
- Paired delta
- 0.33761254290584475
Benchmark result
Significantly WORSE than the tuned baseline (+1497.1%, p=0.0081, wins 0 / 8 paired seeds).
- Benchmark
- Dynamics forecasting
- Model
- rnn_small
- Paired seeds
- 8
- Baseline mean
- 0.0226
- Idea mean
- 0.3602
- Effect (Δ)
- 0.3376 (+1497.1%; negative = idea better)
- Wins
- 0 / 8 paired seeds
- p-value
- 0.0081 (permutation test, 20 000 shuffles)
- Smallest detectable effect
- ±347.2%
- Mechanism
- Not confirmed ✗
- Practical effect
- Hurts
- Baseline tuning
- swept over 3 configs
The final benchmark used the reduced fallback budget of 400 training samples, 200 test samples, and 3 epochs because the original CPU protocol exceeded 40 minutes and the shared CUDA runtime hit a low-level clock assertion. The implementation uses empirical input-cell coverage rather than a learned outer successor envelope or conformal calibration, and long-horizon rollout/MPC control metrics were not tested.
How to run: CUDA_VISIBLE_DEVICES='' OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 /home/maxwelhelp/main/bin/python3 stage2_pac_bench.py
Artifacts
- bench_report.json 3.1 KB View Raw
- pac_transition_cover.py 6.9 KB View Raw
- report.md 1.6 KB View
- report_bench_2026-09-01T115132.md 5.1 KB View
- results.json 6.2 KB View Raw
- stage2_pac_bench.py 7.2 KB View Raw