✗ Failed on benchmark 2026

Adaptive conformal safety margins

Implementation & benchmark of arXiv:2608.10056 — Navigating the Proximity-Safety Balance: Constraint Decomposition for Human Following in Pedestrian Crowds

Usefulness7/10

Difficulty4/10

Novelty5/10

Training Stability Accuracy Generalization World model Rl Training Statistics Probability Optimization

Source paper: Navigating the Proximity-Safety Balance: Constraint Decomposition for Human Following in Pedestrian Crowds arXiv:2608.10056 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Attach an adaptive conformal error radius to every predicted agent and forecast horizon, then use that radius to inflate collision constraints or mask unsafe actions in a learned policy. Unlike a fixed heuristic margin, the radius automatically grows after systematic prediction failures and shrinks when the predictor is accurate, providing an explicit accuracy-versus-conservatism control.

Formulas

$$\delta_{h,k}(t)=\left\|\mathbf{p}_{h}(t)-\mathbf{p}_{h,k}(t-k)\right\|_{2}$$

$$\hat{\delta}^{(m)}_{h,k}(t+1)=\hat{\delta}^{(m)}_{h,k}(t)+\gamma^{(m)}\left(\mathbf{1}\left[\delta_{h,k}(t)>\hat{\delta}^{(m)}_{h,k}(t)\right]-\alpha\right)$$

$$r_{h,k}(t)=Q_{1-\beta}\left(\left\{\hat{\delta}^{(m)}_{h,k}(t)\right\}_{m=1}^{M}\right),\qquad c_{h,k}=\mathbf{1}\!\left[\left\|\hat{\mathbf{p}}_{h,k}-\mathbf{p}_{\mathrm{robot},k}\right\|_2\le R_{\mathrm{robot}}+R_h+r_{h,k}\right]$$

$$\mathcal{L}_{\mathrm{policy}}=\mathcal{L}_{\mathrm{task}}+\sum_{j=1}^{J}\lambda_j\left(\frac{1}{H}\sum_{k=1}^{H}c_{j,k}-\tau_j\right),\qquad \lambda_j\leftarrow\left[\lambda_j+\eta_\lambda\left(\frac{1}{H}\sum_{k=1}^{H}c_{j,k}-\tau_j\right)\right]_+$$

Mathematical statement

The paper defines the k-step prediction error for human h at time t as $\delta_{h,k}(t)=\|\mathbf{p}_{h}(t)-\mathbf{p}_{h,k}(t-k)\|_2$, where $\mathbf{p}_{h}(t)\in\mathbb{R}^2$ is the observed position and $\mathbf{p}_{h,k}(t-k)$ is the position predicted k steps earlier. For each human h, horizon k, and estimator m, adaptive conformal inference maintains a radius $\hat{\delta}^{(m)}_{h,k}(t)$ using $\hat{\delta}^{(m)}_{h,k}(t+1)=\hat{\delta}^{(m)}_{h,k}(t)+\gamma^{(m)}(\mathbf{1}[\delta_{h,k}(t)>\hat{\delta}^{(m)}_{h,k}(t)]-\alpha)$, where $\gamma^{(m)}>0$ is the update step and $\alpha\in(0,1)$ is the target miscoverage rate. The update increases the radius by $\gamma^{(m)}(1-\alpha)$ on an exceedance and decreases it by $\gamma^{(m)}\alpha$ otherwise, so the radius tracks a desired exceedance frequency under distribution shift. In the neural-network adaptation, define a conservative radius $r_{h,k}(t)=Q_{1-\beta}(\{\hat{\delta}^{(m)}_{h,k}(t)\}_{m=1}^M)$, where $Q_{1-\beta}$ is the empirical upper quantile across the M estimators and $\beta$ controls ensemble conservatism. A predicted collision cost is then $c_{h,k}=\mathbf{1}[\|\hat{\mathbf{p}}_{h,k}-\mathbf{p}_{\mathrm{robot},k}\|_2\leq R_{\mathrm{robot}}+R_h+r_{h,k}]$, with robot radius $R_{\mathrm{robot}}$, human radius $R_h$, and predicted robot position $\mathbf{p}_{\mathrm{robot},k}$.

Implementation notes

Integrate the mechanism between the human-motion predictor and the learned action generator. The predictor may be an LSTM, transformer, graph neural network, or diffusion trajectory model producing $\hat{\mathbf p}_{h,k}$ for humans h and horizons k=1,...,H; the policy or planner receives these predictions and the calibrated radii as additional features, while the collision-cost layer uses them to reject or penalize actions. Maintain a tensor radius[h,k,m] and a FIFO buffer of predictions made at previous times. At timestep t: (1) for every delayed forecast with horizon k, retrieve the stored prediction from t-k and compute delta = norm(observed_position[h,t] - stored_prediction[h,k]); (2) for each m, update radius[h,k,m] += gamma[m] * ((delta > radius[h,k,m]) - alpha); optionally clamp radii to nonnegative values; (3) compute r[h,k] as the empirical (1-beta) quantile over m; (4) append the current predictions to the buffer; (5) inflate each collision threshold by r[h,k] and either add the resulting costs to the policy loss or hard-mask actions whose predicted cost exceeds a threshold. If training with constraints, estimate each batch cost and update the nonnegative multiplier lambda[j] using the displayed projected ascent rule. The mathematical quantities are the delayed errors, online radius recursion, and cost inflation; choose gamma[m], alpha, and beta by validation. The first cheap test is a small PPO policy with a pretrained 2D pedestrian predictor in a simulated crossing scenario, compared against fixed margins and uncalibrated predictions. Measure empirical miscoverage, collision rate, target-reaching rate, average path length, and policy-gradient variance. Success means the calibrated system maintains collision probability near the selected budget under shifted pedestrian speeds while losing less proximity or task reward than a fixed conservative margin.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and verified an adaptive conformal radius simulator with nonnegative projection, ensemble upper-quantile support, and a shifted crossing safety comparison. The mechanism manifested: stationary exceedance matched target alpha within 0.00009, while post-shift growth and recovery matched inverse-gamma predictions exactly across all sweeps. In the toy crossing scenario, adaptive margins reduced false-safe frames from 80/80 for a small fixed margin to 4/80 while rejecting 20 safe frames versus 120/120 for the conservative fixed margin; this is promising but not evidence of a learned-policy win.

Agent confidence: 8/10

Baseline: fixed_0.20: 80 false-safe unsafe frames, 0 safe rejections, accept rate 1.00; fixed_0.80: 0 false-safe frames, 120 safe rejections, accept rate 0.00
Idea: adaptive: 4 false-safe unsafe frames, 20 safe rejections, accept rate 0.52, mean radius 0.4528; exceedance absolute error <= 0.00009; ramp and recovery relative errors 0.0; ensemble quantile Q_0.75([0.2,0.4,0.8,1.0])=0.85

Limitations: Only a synthetic one-dimensional crossing proxy was tested; no neural predictor, PPO or policy training, multi-agent trajectories, delayed FIFO prediction buffer, computational-cost measurement, formal finite-sample conformal guarantee, or realistic distribution shifts were evaluated.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.10056) Sep 1, 2026

No significant difference vs. the tuned baseline (+8.2%, p=0.1888, wins 3 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0
Idea mean: 0
Effect (Δ): 0 (+8.2%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.1888 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Tested one built-in matched dynamics task and one adaptive-loss formulation; did not test PPO/action masking, explicit multi-agent collision rollouts, FIFO delayed prediction buffers, ensemble estimator quantiles in the neural benchmark, or broader gamma/alpha/strength sweeps.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented adaptive conformal error-radius weighting as a training-loss intervention on the structurally matched controlled-pendulum dynamics track, using the same rnn_small architecture and paired eight-seed protocol. The trained-model signature confirmed the target exceedance quantitatively (0.197 observed versus alpha=0.2), and recursion sanity checks passed, but the idea had higher test MSE than the swept baseline (2.46837e-05 versus 2.28229e-05; delta=+1.86082e-06, permutation p=0.1888), so there was no significant win.

Baseline: Best lr=0.006; 8-seed mean test MSE=2.282289460708853e-05, std=2.305330446600017e-06
Delta mean: 1.8608170648803934e-06
Idea: Best lr=0.006, alpha=0.2, gamma=0.05, strength=0.5; 8-seed mean test MSE=2.4683711671968922e-05, std=2.8119869264952184e-06
Idea wins: 3
Mechanism signature: map[confirmed:true ramp_observed_vs_predicted_steps:19 vs 19 recovery_observed_vs_predicted_steps:88 vs 88 stationary_abs_error:0.00028000000000000247 target_alpha:0.2 trained_model_retested_exceedance:0.197]
Permutation p value: 0.1888

Benchmark result