# Risk-Calibrated World-Model Gates

- ID: 2482
- Canonical URL: https://synthcore.org/idea/2482/risk-calibrated-world-model-gates
- API JSON: https://synthcore.org/api/idea/2482.json
- API Markdown: https://synthcore.org/api/idea/2482.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.17956](https://arxiv.org/abs/2608.17956)
- Category: theory
- Solves: accuracy, stability, sample-efficiency
- ML areas: world-model, rl, training-dynamics
- Math tags: probability, statistics, control-theory
- Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10

## Idea description

Replace a fixed-size random transition gate with a risk-calibrated gate whose test count is chosen from the estimated probability of a critical event and the cost of shipping a model that misses it. The gate should combine ordinary i.i.d. rollouts with planner-generated probes aimed at high-cost boundaries, because uniform sampling can make a dangerous model appear perfectly accurate.

## Mathematical statement

For a critical event with probability r in one independent gate rollout, the probability that all N rollouts miss it is exactly P_miss=(1-r)^N. To make this probability at most delta, choose N >= log(delta)/log(1-r), equivalently N >= log(1/delta)/(-log(1-r)); for small r this is approximately log(1/delta)/r. If C is the play cost incurred by the shipped model when the gate misses the mode, and C is independent of acceptance, the expected danger risk factorizes as R=C(1-r)^N. If the training sample independently misses the same event with probability (1-r)^M and the acceptance sample has N rollouts, the joint blind-and-accepted probability is (1-r)^(M+N). Here r is the event probability under the gate distribution, N is the number of verification rollouts, M is the number of training samples, delta is the target miss probability, and C can be estimated as planner regret or episode cost under the blind model.

## Key formulas

- $$P(\text{all }N\text{ gate rollouts miss the event})=(1-r)^N$$
- $$N\geq\frac{\log \delta}{\log(1-r)}\approx\frac{\log(1/\delta)}{r}\quad(r\ll1)$$
- $$R=C(1-r)^N$$
- $$P(\text{training miss and gate miss})=(1-r)^M(1-r)^N=(1-r)^{M+N}$$

## Implementation notes

Integrate this at the acceptance stage of a learned simulator or Code World Model, immediately after training and before allowing a planner to use the model. First define an event detector E(s,a,s') for mode-relevant behavior: contact with a boundary, abrupt velocity reset, termination, saturation, collision, or a large disagreement between independently trained predictors. Estimate r under the current gate distribution by counting E, but do not trust the estimate when zero events are observed. Use a conservative lower confidence estimate r_low from a Beta(1,1) prior or a stress-test distribution. If the target miss probability is delta, set N=ceil(log(delta)/log(1-r_low)). Run N ordinary rollouts and K planner-directed rollouts initialized near the highest predicted value-gradient, uncertainty, or boundary score. Reject the model if any transition violates the reference environment or if the estimated high-cost risk C_hat*(1-r_low)^N exceeds a deployment threshold. Pseudocode: sample or generate states z_i; execute reference and learned transitions; compute E_i and costs C_i; update r_low; compute N_required; accept only when N>=N_required and C_hat*(1-r_low)^N<=R_max. The exact mathematics supplies the miss probability and required sample count; r_low, C_hat, and the stress distribution are empirical. First test on a cart-with-wall or small MuJoCo contact task using an MLP simulator and random-gate baseline. Measure accepted blind models, contact detection rate, false rejection rate, and planner regret. Success is fewer catastrophic accepted models at the same verification budget, or a large regret reduction for a modest rollout increase.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a reproducible risk-calibrated gate MVP with required-rollout sizing, conservative lower event bounds, ordinary sampling, and planner-directed probes. The exact miss law matched Monte Carlo within 0.00272 absolute error; all required-count sweeps achieved target delta=0.05, with the small-r approximation within 3.2%. Matched-budget probes reduced observed miss probability from 0.77877 to 0.03343 at r=0.005 and from 0.60664 to 0.00065 at r=0.01, closely matching the multiplicative prediction. This confirms the mechanism, but not a real deployment win because probe effectiveness was prescribed rather than learned.

### Mechanism check

- Verdict: Built a reproducible risk-calibrated gate MVP with required-rollout sizing, conservative lower event bounds, ordinary sampling, and planner-directed probes. The exact miss law matched Monte Carlo within 0.00272 absolute error; all required-count sweeps achieved target delta=0.05, with the small-r approximation within 3.2%. Matched-budget probes reduced observed miss probability from 0.77877 to 0.03343 at r=0.005 and from 0.60664 to 0.00065 at r=0.01, closely matching the multiplicative prediction. This confirms the mechanism, but not a real deployment win because probe effectiveness was prescribed rather than learned.
- Confidence: 8/10
- Limitations: No learned simulator, reference environment, planner regret, collision detector, deployment threshold, or false-rejection analysis was tested. The stress probe rate q=min(0.60,30r) is synthetic and assumes independent stationary events.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.164
- Paired wins: 3/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/958/bench_report.json)
- [official_stage2.py](https://synthcore.org/code/958/official_stage2.py)
- [report.md](https://synthcore.org/code/958/report.md)
- [report_bench_2026-09-01T171258.md](https://synthcore.org/code/958/report_bench_2026-09-01T171258.md)
- [results.json](https://synthcore.org/code/958/results.json)
- [risk_gate_experiment.py](https://synthcore.org/code/958/risk_gate_experiment.py)
- [stage2_bench.py](https://synthcore.org/code/958/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/958)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.