✓✓ Beats tuned baseline 2026

Directional Conformal Residual Sets for Neural Dynamics

Implementation & benchmark of arXiv:2607.29344 — Directional Conformal Uncertainty Quantification from Learned Model Discrepancy

Usefulness8/10
Difficulty4/10
Novelty6/10

Source paper: Directional Conformal Uncertainty Quantification from Learned Model Discrepancy arXiv:2607.29344 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Augment a neural dynamics model with a separately trained discrepancy predictor and calibrate an asymmetric conformal residual score. Use the resulting state- and input-dependent uncertainty set to reject, damp, or regularize neural rollouts when they leave a calibrated region, rather than treating all residual directions as equally uncertain.

Formulas

$$e_i=y_i-f_\theta(z_i),\qquad S_i=\max\left\{0,\left|e_i\right|-\operatorname{sign}(e_i)d_\phi(z_i)\right\}.$$
$$q=\operatorname{Quantile}_{\lceil(N+1)(1-\alpha)\rceil/N}\left(S_1,\ldots,S_N\right),\qquad \tilde q(z)=\max\{q,|d_\phi(z)|\}.$$
$$C(z)=\left\{y:\max\left(0,|y-f_\theta(z)|-\operatorname{sign}(y-f_\theta(z))d_\phi(z)\right)\leq\tilde q(z)\right\},\qquad \Pr(y\in C(z))\geq1-\alpha.$$
$$C(z)=[d_\phi(z)-\tilde q(z),\ d_\phi(z)+\tilde q(z)]\quad\text{when }\tilde q(z)\geq|d_\phi(z)|;\qquad |C(z)|=2\tilde q(z).$$

Mathematical statement

Let the observed transition be $y_i=x_{i+1}$, the nominal neural predictor be $f_\theta(z_i)$ with $z_i=(x_i,u_i)$, and the learned signed discrepancy be $d_\phi(z_i)$ in the scalar-output case. Define the residual $e_i=y_i-f_\theta(z_i)$ and directional score $S_i=\max\{0,|e_i|-\operatorname{sign}(e_i)d_\phi(z_i)\}$. A positive discrepancy reduces the score for positive residuals and increases it for negative residuals; the opposite occurs for negative discrepancy. Given $N$ exchangeable calibration examples and miscoverage level $\alpha$, let $q$ be the $\lceil(N+1)(1-\alpha)\rceil$-th split-conformal order statistic of the scores. Define $\tilde q(z)=\max\{q,|d_\phi(z)|\}$ to guarantee that the interval contains zero, and return $C(z)=\{y:S(y-f_\theta(z),z)\leq\tilde q(z)\}$. Under exchangeability, the marginal coverage satisfies $\Pr[y\in C(z)]\geq1-\alpha$; the useful regime is where the directional score quantile is smaller than the symmetric residual quantile.

Implementation notes

1. Integration point: use this as an uncertainty head around a neural one-step dynamics model in a world model, model-based RL agent, or neural MPC system. Split transition data into three disjoint sets: fit $f_\theta$ on the first set, fit the discrepancy network $d_\phi(z)$ to predict the signed residual on the second set, and perform conformal calibration only on the third set. Freeze both networks and the calibration quantile during evaluation. For vector states, apply the scalar construction componentwise initially; a later implementation can replace absolute value by a Minkowski gauge for joint sets. 2. Pseudocode: compute $e_i=y_i-f_\theta(z_i)$ on calibration data; compute $S_i=\max(0,|e_i|-\operatorname{sign}(e_i)d_\phi(z_i))$; sort scores; select index $k=\lceil(N+1)(1-\alpha)\rceil$; at inference compute $d=d_\phi(z)$ and $q_z=\max(q,|d|)$; return the interval $[d-q_z,d+q_z]$. During rollout, calculate normalized violation $v_t=\max(0,|\hat x_{t+1}-f_\theta(z_t)|-q_{z_t})$ and either terminate the rollout, increase MPC constraint tightening, or multiply the training loss by $1+\lambda v_t$. 3. Computed from the paper: the directional score, order-statistic calibration, exchangeability-based coverage guarantee, and origin-containing interval construction. Estimated empirically: the discrepancy network, local coverage, violation frequency, and whether the calibration distribution remains exchangeable under policy shift. 4. First cheap experiment: train a one-step MLP dynamics model on CartPole, Pendulum, or a low-dimensional MuJoCo transition dataset. Compare symmetric split-conformal residual intervals with directional intervals at $\alpha=0.1$ over 10-step open-loop rollouts. Measure marginal coverage, conditional coverage in bins of $z$, average interval width, and rollout divergence. The quantitative prediction is coverage at least approximately $90\%$ on IID test transitions for both methods, while directional width should shrink in regions where the discrepancy predictor captures a persistent signed bias; specifically, if the symmetric residual 90th percentile is $q_0$ and the directional score percentile is $q_d$, the mean width should approach $2q_d$ and satisfy $q_d<q_0$ by at least 10% in high-bias regions. Coverage failure under policy shift should appear before width improvement disappears, providing a falsifiable calibration diagnostic.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built directional conformal scalar intervals with split calibration and a separate MLP discrepancy head. The toy mechanism manifested: with signed bias 0.8, the directional score quantile decreased approximately linearly as q≈q0−rho·0.8 for rho≤0.5, coverage stayed near 90.1%, and the zero-containing-radius floor activated between rho=0.5 and 0.75, after which width increased. The neural experiment did not show a practical width win: directional coverage was comparable but its mean width was slightly larger.

Agent confidence
8/10
Baseline
Oracle symmetric width 2.2391 and coverage 0.9011; neural symmetric width 0.4225 and coverage 0.9060.
Idea
Oracle directional width 1.4391 at rho=0.5, 64.3% of baseline, with coverage 0.9011; at rho=0.75 width 1.2000 and coverage 0.9452. Neural directional width 0.4435 and coverage 0.9098; discrepancy RMSE 0.2789.

Limitations: Only scalar synthetic one-step transitions were tested. CartPole, multi-step rollout rejection or damping, vector-valued joint sets, policy distribution shift, repeated seeds, runtime comparisons, and optimized discrepancy-head training were not evaluated.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.29344)

Beats the tuned baseline by 21.5% (p=0.0163, wins 7 / 8 paired seeds; metric: lower is better).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0006
Idea mean
0.0005
Effect (Δ)
-0.0001 (−21.5%; negative = idea better)
Wins
7 / 8 paired seeds
p-value
0.0163 (permutation test, 20 000 shuffles)
Smallest detectable effect
±16.2%
Mechanism
Confirmed ✓
Practical effect
Helps
Baseline tuning
swept over 3 configs
Limitations:

Only the registered actuated-pendulum dynamics track was tested. Policy-shift calibration, multi-step rollout rejection or damping, vector-valued joint sets, runtime/FLOP accounting, and larger datasets were not tested. The discrepancy head used the shared learning-rate/epoch grid rather than a separate larger hyperparameter search; empirical coverage was slightly below the nominal 90% level.

How to run: python3 bench_directional.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark latest Worked ✓

Agent confidence: 8/10

Directional Conformal Residual Sets were implemented as a separate discrepancy GRU head with directional conformal calibration and signed prediction correction on the registered actuated-pendulum dynamics track. Against the tuned canonical rnn_small baseline, the idea achieved a significant paired MSE improvement: delta_mean=-0.00012459131539799273, 7/8 wins, permutation p=0.0163. The trained-model mechanism signature was confirmed: mean directional score quantile was 0.02553695 versus symmetric 0.03202619, ratio 0.8301, although finite-test interval coverage averaged 0.8828.

Baseline
lr=0.01, epochs=8; full 8-seed mean MSE=0.0005782862572232261, std=0.0001567340024827472.
Idea
lr=0.01, epochs=8; full 8-seed mean MSE=0.0004536949418252334, std=0.00012573084952527102; paired delta=-0.00012459131539799273; p=0.0163.

Benchmark result

Beats the tuned baseline by 21.5% (p=0.0163, wins 7 / 8 paired seeds; metric: lower is better).

Benchmark
Dynamics forecasting
Model
rnn_small
Paired seeds
8
Baseline mean
0.0006
Idea mean
0.0005
Effect (Δ)
-0.0001 (−21.5%; negative = idea better)
Wins
7 / 8 paired seeds
p-value
0.0163 (permutation test, 20 000 shuffles)
Smallest detectable effect
±16.2%
Mechanism
Confirmed ✓
Practical effect
Helps
Baseline tuning
swept over 3 configs
Limitations:

Only the registered actuated-pendulum dynamics track was tested. Policy-shift calibration, multi-step rollout rejection or damping, vector-valued joint sets, runtime/FLOP accounting, and larger datasets were not tested. The discrepancy head used the shared learning-rate/epoch grid rather than a separate larger hyperparameter search; empirical coverage was slightly below the nominal 90% level.

How to run: python3 bench_directional.py

Artifacts

Implementation overview ⬇ Download all as ZIP 7 files · code, reports and structured results