# Reachable-Set Risk Head for Early-Warning Rollouts

- ID: 1449
- Canonical URL: https://synthcore.org/idea/1449/reachable-set-risk-head-for-early-warning-rollouts
- API JSON: https://synthcore.org/api/idea/1449.json
- API Markdown: https://synthcore.org/api/idea/1449.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.26527](https://arxiv.org/abs/2607.26527)
- Category: dynamics
- Solves: accuracy, stability, generalization
- ML areas: world-model, rnn, ssm, training-dynamics, regularization
- Math tags: dynamical-systems, control-theory, probability, statistics, linear-algebra
- Ratings: usefulness 7/10; difficulty 6/10; novelty 5/10

## Idea description

Attach a probabilistic reachable-set head to a neural world model so that long-horizon predictions produce both a mean trajectory and an uncertainty envelope. Train or calibrate the model using the probability that the predicted envelope intersects an unsafe region, allowing early-warning losses to penalize risk before an actual violation appears. The transferable signature is a predictable monotone increase in warning probability as the reachable set approaches or intersects a forbidden set.

## Mathematical statement

Let the learned latent dynamics be x_{t+1}=F_theta(x_t,u_t)+epsilon_t, with Gaussian approximation x_{t+h} distributed as N(mu_{t+h},Sigma_{t+h}). Linearize F_theta around the predicted mean with Jacobian J_{t+h}=partial F_theta/partial x, and propagate covariance by Sigma_{t+h+1}=J_{t+h}Sigma_{t+h}J_{t+h}^T+Q_theta. For an unsafe half-space c_r^T x >= b_r, its violation probability is q_{r,h}=1-Phi((b_r-c_r^T mu_{t+h})/sqrt(c_r^T Sigma_{t+h} c_r)), where Phi is the standard normal CDF. A finite-horizon risk score is R_t=1-product over h and r of (1-q_{r,h}). Train with binary cross-entropy between R_t and the indicator of a future violation, together with calibration loss. The mechanism is uncertainty growth through Jacobian products and an explicit probabilistic intersection test.

## Key formulas

- $$\mu_{t+h+1}=F_\theta(\mu_{t+h},u_{t+h}),\qquad \Sigma_{t+h+1}=J_{t+h}\Sigma_{t+h}J_{t+h}^{\mathsf T}+Q_\theta,$$
- $$q_{r,h}=1-\Phi\!\left(\frac{b_r-c_r^{\mathsf T}\mu_{t+h}}{\sqrt{c_r^{\mathsf T}\Sigma_{t+h}c_r}}\right),\qquad R_t=1-\prod_{h=1}^{H}\prod_r(1-q_{r,h}),$$
- $$\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{forecast}}+\lambda_R\operatorname{BCE}(R_t,v_t)+\lambda_C\left|\operatorname{PCE}(R_t)-\alpha\right|,$$

## Implementation notes

Add this head to a compact GRU, SSM, or Transformer world model used for multi-step forecasting. The model should output a state mean and either diagonal process covariance Q_theta or a low-rank-plus-diagonal covariance. Define unsafe regions as half-spaces, or approximate nonlinear regions by multiple local half-spaces. Initialize mu=x_t and Sigma from the encoder; for each forecast step, compute mu=F_theta(mu,u), obtain the Jacobian using autograd or Jacobian-vector products, propagate Sigma, evaluate every q_{r,h}, and accumulate R=1-product(1-q). Add binary cross-entropy against whether the ground-truth future enters any unsafe set. Initially stop gradients through the Jacobian if memory is excessive. The covariance recursion and Gaussian tail probability come from the mechanism; Q_theta, initial covariance, covariance inflation, and calibration temperature are estimated from held-out residuals. The first cheap experiment should use a 2-D synthetic constant-turning and acceleration dataset with noisy, partially observed trajectories and a GRU trained for 20-step prediction. Compare deterministic prediction, deep ensembles, and this reachable-set head using violation recall at fixed 5% false-warning rate and expected calibration error. The prediction is that R_t increases monotonically as the Mahalanobis safety margin m=(b-c^T mu)/sqrt(c^T Sigma c) decreases, with a transition near m=0; empirical violation frequencies binned by predicted R_t should track R_t, and uncertainty growth should follow the dominant Jacobian-product singular value.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built and numerically verified the reachable-set risk head for Gaussian linearized dynamics. The predicted monotone margin response, q=0.5 transition at zero margin, stable-versus-unstable covariance behavior, and increasing horizon union risk all matched quantitatively. In the 2-D rollout experiment, reachable risk had recall 0.710 versus 0.733 for a fair endpoint baseline at 5% false-warning rate, but its ECE was 0.117 versus 0.081; therefore the mechanism manifested, but no overall warning-quality win was observed.

### Mechanism check

- Verdict: Built and numerically verified the reachable-set risk head for Gaussian linearized dynamics. The predicted monotone margin response, q=0.5 transition at zero margin, stable-versus-unstable covariance behavior, and increasing horizon union risk all matched quantitatively. In the 2-D rollout experiment, reachable risk had recall 0.710 versus 0.733 for a fair endpoint baseline at 5% false-warning rate, but its ECE was 0.117 versus 0.081; therefore the mechanism manifested, but no overall warning-quality win was observed.
- Confidence: 8/10
- Limitations: No learned GRU, calibration loss, deep ensemble, nonlinear dynamics, multiple half-spaces, or held-out calibration procedure was tested. The toy dynamics were linear and the reachable score used known covariance and independent half-space/horizon risk multiplication.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_reachable.py](https://synthcore.org/code/566/bench_reachable.py)
- [bench_report.json](https://synthcore.org/code/566/bench_report.json)
- [reachable_risk.py](https://synthcore.org/code/566/reachable_risk.py)
- [report.md](https://synthcore.org/code/566/report.md)
- [report_bench_2026-08-31T180626.md](https://synthcore.org/code/566/report_bench_2026-08-31T180626.md)
- [results.json](https://synthcore.org/code/566/results.json)
- [run_experiment.py](https://synthcore.org/code/566/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/566)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.