△ Mechanism confirmed, baseline not beaten 2026

Reachset-Conformance Noise Calibration

Implementation & benchmark of arXiv:2608.02308 — A General Set-Based Framework for Cognitive State Estimation: Theory and Application to Conditionally Automated Driving

Usefulness7/10

Difficulty5/10

Novelty8/10

Regularization Stability Generalization Accuracy Rnn Ssm World model Regularization Training dynamics Control theory Dynamical systems Optimization Probability

Source paper: A General Set-Based Framework for Cognitive State Estimation: Theory and Application to Conditionally Automated Driving arXiv:2608.02308 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Calibrate process and observation uncertainty bounds by requiring a learned neural dynamical model to contain calibration trajectories in its reachable sets, instead of fitting a Gaussian noise model. The resulting bounds can control an uncertainty-aware loss, trigger teacher forcing or re-observation, and identify latent coordinates whose dynamics are not adequately modeled. This transfers the paper's conformance principle into a falsifiable training monitor and adaptive rollout schedule.

Formulas

$$r^x_k=x_{k+1}-\hat f_\theta(x_k,u_k),\qquad r^y_k=y_k-\hat h_\theta(x_k).$$

$$\epsilon_{x,i}=\max_{k\in\mathcal{D}_{\mathrm{cal}}}|r^x_{k,i}|,\qquad \epsilon_{y,j}=\max_{k\in\mathcal{D}_{\mathrm{cal}}}|r^y_{k,j}|.$$

$$\mathcal{X}_{k+1|k}=\hat f_\theta(\mathcal{X}_k,u_k)\oplus[-\epsilon_x,\epsilon_x],\qquad \mathcal{C}_{k+1}=\mathcal{X}_{k+1|k}\cap\{x:|\hat h_\theta(x)-y_{k+1}|\leq\epsilon_y\}.$$

$$\widehat p_{\mathrm{violation}}=\frac{1}{M}\sum_{m=1}^{M}\mathbf{1}\{\mathcal{C}^{(m)}_{k}=\varnothing\},\qquad \widehat p_{\mathrm{violation}}\approx\delta\text{ under calibrated bounds.}$$

Mathematical statement

Let $\hat f_\theta$ be a learned latent transition map, $\hat h_\theta$ its decoder, and let $x_k$ be an encoded latent state. Define one-step residuals $r^x_k=x_{k+1}-\hat f_\theta(x_k,u_k)$ and observation residuals $r^y_k=y_k-\hat h_\theta(x_k)$. For componentwise bounded uncertainty, choose the smallest nonnegative bounds $\epsilon_x,\epsilon_y$ such that $|r^x_k|\leq\epsilon_x$ and $|r^y_k|\leq\epsilon_y$ for every calibration trajectory, optionally allowing an outlier fraction $\delta$. Equivalently, the conformant bounds are componentwise maxima, $\epsilon_{x,i}=\max_k|r^x_{k,i}|$ and $\epsilon_{y,j}=\max_k|r^y_{k,j}|$, or empirical $(1-\delta)$ quantiles for contaminated data. A rollout is conformant when its reachable set generated with these bounds intersects every observation-consistency set. The mechanism predicts that the first violation rate on held-out data is approximately the calibration outlier rate $\delta$; systematic violation growth indicates model mismatch rather than merely stochastic uncertainty.

Implementation notes

1. Integration point: add a calibration phase after ordinary teacher-forced training of a GRU, neural ODE discretization, or latent SSM. Store encoded states, transitions, and decoder outputs on a held-out calibration split. 2. Pseudocode: compute rx=x_next-f_theta(x,u) and ry=y-h_theta(x); set eps_x=quantile(abs(rx),1-delta) and eps_y=quantile(abs(ry),1-delta) componentwise; during rollout propagate an interval or zonotope using these bounds; after each observed target, test whether the predicted output interval intersects [y-eps_y,y+eps_y]; if not, increase teacher-forcing frequency, reduce rollout length, or apply a conformant-loss penalty relu(dist(y,h(X))-eps_y). Recalibrate periodically as the model changes. 3. Computed directly: residual bounds and the nonempty reachset-consistency test. Estimated empirically: quantiles, latent encoder uncertainty, and the nonlinear interval/Jacobian bound for $f_\theta(\mathcal{X})$. Use a conservative Lipschitz bound or sampled corners for the latter. 4. First cheap experiment: train a GRU and this calibrated variant on noisy sine-wave and Lorenz-63 forecasting with bounded uniform noise, then test on a shifted disturbance amplitude. Sweep $\delta\in\{0,0.01,0.05,0.1\}$. The predicted signature is held-out empty-intersection frequency near $\delta$ on in-distribution data, while it rises sharply under disturbance amplitudes exceeding the calibrated bound. Plot violation rate and forecast RMSE versus rollout horizon; the conformant model should show an early-warning kink at the horizon where reachable output width first exceeds the observation tolerance, rather than silently diverging.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a scalar latent dynamical-system MVP implementing componentwise residual quantile calibration and one-step reachable-set/observation-consistency testing. Held-out latent and observation violation rates matched delta within 0.002, calibrated bounds scaled linearly with process amplitude, and shifted-disturbance reachset violations rose sharply from 0.003 at 1x to 0.489 at 3x. The direct empty-intersection rate is lower than delta in-distribution because latent and observation bounds are Minkowski-added, making that test conservative.

Agent confidence: 9/10

Baseline: Gaussian mean+1.645*std bounds: shifted reachset-empty rate 0.000 at 1x, 0.016 at 2x, and 0.218 at 3x; latent violation rates were 0.000, 0.273, and 0.522.
Idea: For delta={0.01,0.05,0.10}, latent/observation violation pairs were (0.0106,0.0119), (0.0517,0.0512), and (0.0991,0.1000). For process amplitudes {0.04,0.08,0.16}, eps_x was {0.0380,0.0760,0.1518}, with eps_x/amplitude approximately 0.95. Shifted reachset-empty rates at multipliers {1,1.5,2,3} were {0.0025,0.0914,0.2331,0.4895}.

Limitations: Only a scalar linear identity-decoder system was tested; no GRU, Lorenz-63, nonlinear interval propagation, adaptive teacher forcing, conformant-loss training, multidimensional latent coordinates, or long-horizon forecasting was evaluated. The Gaussian comparison is a simple residual-width baseline rather than a tuned probabilistic model.

How to run: python3 reachset_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.02308) Sep 2, 2026

Significantly WORSE than the tuned baseline (+13018.1%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0042
Idea mean: 0.5528
Effect (Δ): 0.5486 (+13018.1%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 0 configs

Limitations:

Only the built-in dynamics track was tested; full latent encoder-decoder reachability, nonlinear interval propagation, adaptive teacher forcing, shifted disturbances, and broader sweeps were not tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

The dynamics benchmark implemented residual-quantile conformant-loss calibration with rnn_small. Coverage behavior was confirmed, but the idea significantly lost on independent test MSE, so there is no demonstrated training win.

Baseline: Test MSE mean 0.00421384; best lr=0.01, epochs=8, weight_decay=0.0001.
Idea: Test MSE mean 0.55277710; paired delta +0.54856325; 0/8 wins; permutation p=0.0081. Signature confirmed=true with target violation 0.05 and observed violation 0.00833333.

Benchmark result