✗ Failed on benchmark 2026

Conformal Residual Certificates for Neural Rollouts

Implementation & benchmark of arXiv:2608.06419 — Certified Feedforward Tracking for Unknown Nonlinear Systems via Invertible Neural Networks

Usefulness7/10

Difficulty4/10

Novelty6/10

Theory Stability Generalization World model Ssm Regularization Inference speedup Statistics Dynamical systems Control theory Linear algebra

Source paper: Certified Feedforward Tracking for Unknown Nonlinear Systems via Invertible Neural Networks arXiv:2608.06419 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Attach a finite-sample conformal error radius to a neural surrogate of a dynamical or sequence model, and propagate that radius through the model's local sensitivity. The model should expose a calibrated prediction set or abstain whenever the accumulated bound exceeds a task-specific tolerance, making long-horizon failure a measurable coverage event rather than an unobserved drift.

Formulas

r_i = ||y_i - F_hat(z_i)||_2, and q_(1-alpha) = Quantile_rank(r_1,...,r_n_cal), where rank = ceil((n_cal + 1)(1 - alpha))

P(||F_s(z) - F_hat(z)||_2 <= q_(1-alpha)) >= 1 - alpha

R_H = q_(1-alpha) * sum_{t=0}^{H-1} product_{j=t+1}^{H-1} L_j <= q_(1-alpha) * (L^H - 1)/(L - 1)

L_j = ||partial F_hat / partial x evaluated at (x_j,u_j)||_2

Mathematical statement

Let z denote an input window or Fourier-coefficient vector, F_s(z) the unknown true input-output map, and F_hat(z) the trained neural surrogate. On a calibration set of n_cal examples, define nonconformity scores r_i = ||y_i - F_hat(z_i)||_2. For miscoverage level alpha, let q_(1-alpha) be the conformal quantile with the finite-sample rank correction. Then the marginal guarantee is P(||F_s(z) - F_hat(z)||_2 <= q_(1-alpha)) >= 1 - alpha. If a downstream rollout has local Lipschitz sensitivities L_t, a conservative horizon-H radius is R_H = q_(1-alpha) times the sum from t = 0 to H - 1 of the products of later L_j values. When every L_j <= L, this becomes q_(1-alpha)(L^H - 1)/(L - 1), or Hq_(1-alpha) when L = 1. The transferable mechanism is calibrated residual coverage combined with geometric error amplification.

Implementation notes

First, integrate the method after training any one-step or block-prediction model F_hat_theta(x_t,u_t), including an MLP, GRU, Transformer dynamics model, or Fourier-domain sequence model. Reserve temporally separated calibration trajectories and do not use them for gradient updates. Second, implement the following procedure: fit theta on training trajectories; compute r_i = norm(x_(i,t+1) - F_hat_theta(x_(i,t),u_(i,t))) on calibration windows; choose q using the corrected conformal order statistic; during a rollout, estimate each Jacobian spectral norm L_t with one or two power iterations through automatic differentiation; return the prediction and R_H = q times the sum of products of later gains. Trigger abstention, replanning, a shorter rollout, or a fallback model if R_H exceeds a task tolerance epsilon. The paper supplies the residual-quantile and coverage mechanism; the implementation must empirically estimate Jacobian gains, temporal dependence effects, and whether one-step coverage transfers to trajectories. The first cheap experiment should use a small MLP or GRU on a damped pendulum or synthetic Duffing oscillator, with alpha equal to 0.05 or 0.1, compared against an uncertified rollout. The predicted signature is one-step held-out coverage at least 1 - alpha under exchangeable calibration data and a rollout radius following the geometric law R_H proportional to (L^H - 1)/(L - 1). Measure the horizon at which R_H crosses epsilon and test whether the empirical crossing is within 20 percent of the predicted crossing.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reproducible conformal-residual certificate for a learned one-step dynamical surrogate, including corrected finite-sample quantiles, autodiff local gains, radius propagation, and abstention. The mechanism manifested: Monte Carlo one-step coverage was 0.899–0.905 for target 0.90 across calibration sizes, direct and closed-form geometric radii agreed to machine precision, and scalar recurrence crossing horizons matched predictions exactly for L=1.0, 1.1, and 1.3. In the learned rollout, one-step and all-horizon certificate coverage were both 1.0, but the baseline had zero final tolerance violations and the certificate abstained around horizon 10.9, demonstrating conservatism rather than a performance win.

Agent confidence: 8/10

Baseline: Learned MLP rollout: mean final absolute error 0.01994; final error > tolerance(0.20) rate 0.0
Idea: Calibration q=0.01855; one-step coverage 1.0 vs 0.90 target; all-horizon certificate coverage 1.0; mean final radius 0.1990; mean abstention horizon 10.91; mean local gain 0.8849. Math sweeps: coverage 0.8991–0.9053; geometric relative error <=1.3e-16; crossing predictions exact where finite.

Limitations: The dynamical example is a one-dimensional synthetic stable map rather than a pendulum/Duffing trajectory dataset; calibration/test inputs are i.i.d. rather than temporally dependent trajectories. The scalar crossing sweep uses the certificate recurrence itself, so it validates the formula rather than independently validating distribution-free long-horizon coverage. No FLOP/speed or replanning comparison was tested, and the reported certificate is conservative because local gains are estimated along the predicted path while q is reused at every step.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.06419) Sep 2, 2026

No significant difference vs. the tuned baseline (−-0.0%, p=1, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0014
Idea mean: 0.0014
Effect (Δ): 0 (−-0.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 0 configs

Limitations:

No changed abstention, replanning, or fallback policy was evaluated with a downstream utility metric. Calibration used a held-out suffix of windows rather than explicit trajectory IDs, and sensitivity validation used directional finite differences rather than full spectral power iteration.

How to run: python3 bench_conformal.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Built conformal residual certificates for the matched dynamics track using rnn_small and the canonical train_model path. Across 8 paired seeds, baseline and idea test MSE were identical because certification was inference-time metadata, yielding no measurable effect. Coverage was 0.8981 versus the 0.90 target and Jacobian sensitivity did not agree with finite differences, so the mechanism was not confirmed.

Baseline: Best lr=0.003, epochs=20; 8-seed test MSE mean 0.00138030, std 0.00046111
Idea: lr=0.003, epochs=20; 8-seed test MSE mean 0.00138030, std 0.00046111; paired delta 0.0; permutation p=1.0; q=0.055293; coverage=0.898125; Jacobian gain=0.394522; finite-difference gain=0.063367

Benchmark result