# Criticality-Guided Failure Replay

- ID: 1524
- Canonical URL: https://synthcore.org/idea/1524/criticality-guided-failure-replay
- API JSON: https://synthcore.org/api/idea/1524.json
- API Markdown: https://synthcore.org/api/idea/1524.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.28251](https://arxiv.org/abs/2607.28251)
- Category: training
- Solves: sample-efficiency, accuracy, generalization
- ML areas: optimizer, training-dynamics, rl, data-augmentation
- Math tags: probability, statistics, optimization, dynamical-systems
- Ratings: usefulness 7/10; difficulty 4/10; novelty 5/10

## Idea description

Train a lightweight auxiliary predictor C_phi(s) for the probability that the current policy will eventually fail from state s, then bias environment resets, replay sampling, or data replacement toward high-criticality states. Correct the resulting policy-training samples with importance weights so the expected gradient still targets the original data distribution rather than an uncontrolled failure-only objective.

## Mathematical statement

Let s be a state or initial condition, y in {0,1} indicate eventual rollout failure, and C_phi(s) in (0,1) estimate P(y=1|s). Fit phi by binary cross-entropy. Given a target distribution p(s), define a criticality-biased proposal q_alpha(s) proportional to p(s) times [epsilon + C_phi(s)]^alpha, where alpha >= 0 controls enrichment and epsilon > 0 prevents zero probability. A sample drawn from q_alpha receives importance weight w(s) = p(s)/q_alpha(s), or a self-normalized version. If q_alpha has support wherever p is nonzero, the weighted expectation equals the original policy objective; clipping weights reduces variance at the cost of bias.

## Key formulas

- $$\mathcal{L}_{\mathrm{crit}}(\phi)=-\mathbb{E}_{(s,y)}\left[y\log C_{\phi}(s)+(1-y)\log\left(1-C_{\phi}(s)\right)\right],\qquad C_{\phi}(s)\approx \Pr(y=1\mid s).$$
- $$q_{\alpha}(s)=\frac{p(s)\,[\epsilon+C_{\phi}(s)]^{\alpha}}{Z_{\alpha}},\qquad Z_{\alpha}=\mathbb{E}_{p}\left[[\epsilon+C_{\phi}(s)]^{\alpha}\right].$$
- $$w(s)=\frac{p(s)}{q_{\alpha}(s)}=\frac{Z_{\alpha}}{[\epsilon+C_{\phi}(s)]^{\alpha}},\qquad \mathbb{E}_{q_{\alpha}}[w(s)\ell_{\theta}(s)]=\mathbb{E}_{p}[\ell_{\theta}(s)].$$
- $$\mathrm{ESS}=\frac{\left(\sum_{i=1}^{B}w_i\right)^2}{\sum_{i=1}^{B}w_i^2},\qquad \rho_{\mathrm{fail}}=\frac{\mathbb{E}_{q_{\alpha}}[y]}{\mathbb{E}_{p}[y]}.$

## Implementation notes

1. Integration point: add a small MLP criticality head C_phi to the data pipeline, not to the policy action output. For RL, attach each visited state to the eventual episode-failure label; for imitation or supervised policy learning, attach the reset state or scene configuration to the success/failure outcome. Maintain a rolling criticality buffer containing failed and successful rollouts, plus a policy-training buffer containing state/action or state/return supervision. 2. Pseudocode: collect ordinary rollouts; append (s_t,y) to the criticality buffer; update phi with BCE; score candidate states with c_i=sigmoid(C_phi(s_i)); set a_i=(epsilon+c_i)^alpha; sample indices with probability q_i proportional to the original pool probability p_i times a_i; compute r_i=p_i/q_i; optionally clip r_i to [0,w_max] and normalize by the batch mean; update theta using the weighted loss sum_i r_i ell_theta(s_i). Refresh scores periodically and reduce alpha if ESS/B falls below a threshold. 3. Computed from the paper: BCE criticality learning, policy-outcome labels, failure-focused replacement, and importance-weighted resampling. Estimated empirically: calibration of C_phi, q_i, normalization, and the effect of clipping. 4. First cheap experiment: train an actor-critic or behavior-cloning MLP on CartPole with injected initial-state and action perturbations. Compare uniform replay against alpha in {0.5,1,2}, with and without importance weighting. The predicted signatures are monotonic growth of sampled failure fraction with alpha, decreasing ESS/B, and better agreement between weighted-batch and large-uniform-batch gradients than between unweighted-biased and uniform gradients. The measured enrichment ratio should match the q_alpha prediction within 20%; use the largest alpha with ESS/B above 0.3 and weighted-gradient cosine similarity above 0.9.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a finite-population criticality replay MVP with a trained auxiliary failure predictor, proposal sampling, importance weighting, ESS calculation, and replay training. Failure enrichment matched the q_alpha formula within 0.6% across alpha 0.5–4, ESS decreased sharply with alpha, and the exact importance-weighted gradient identity held to 2.3e-14 L2 error. Corrected replay matched uniform replay at alpha 0.5 and 1.0, while uncorrected replay degraded substantially; alpha 2 showed variance-related deterioration consistent with its low ESS.

### Mechanism check

- Verdict: Built a finite-population criticality replay MVP with a trained auxiliary failure predictor, proposal sampling, importance weighting, ESS calculation, and replay training. Failure enrichment matched the q_alpha formula within 0.6% across alpha 0.5–4, ESS decreased sharply with alpha, and the exact importance-weighted gradient identity held to 2.3e-14 L2 error. Corrected replay matched uniform replay at alpha 0.5 and 1.0, while uncorrected replay degraded substantially; alpha 2 showed variance-related deterioration consistent with its low ESS.
- Confidence: 9/10
- Limitations: This is a controlled finite-state regression replay test rather than a full CartPole actor-critic experiment. The critic is logistic rather than an MLP, clipping and self-normalized weights were not benchmarked, and real-environment wall-clock or sample-efficiency gains were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.24735
- Paired wins: 2/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/593/bench_report.json)
- [criticality_replay_experiment.py](https://synthcore.org/code/593/criticality_replay_experiment.py)
- [report.md](https://synthcore.org/code/593/report.md)
- [report_bench_2026-08-31T192409.md](https://synthcore.org/code/593/report_bench_2026-08-31T192409.md)
- [results.json](https://synthcore.org/code/593/results.json)
- [stage2_bench.py](https://synthcore.org/code/593/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/593)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.