Criticality-Guided Failure Replay

1. Integration point: add a small MLP criticality head C_phi to the data pipeline, not to the policy action output. For RL, attach each visited state to the eventual episode-failure label; for imitation or supervised policy learning, attach the reset state or scene configuration to the success/failure outcome. Maintain a rolling criticality buffer containing failed and successful rollouts, plus a policy-training buffer containing state/action or state/return supervision. 2. Pseudocode: collect ordinary rollouts; append (s_t,y) to the criticality buffer; update phi with BCE; score candidate states with c_i=sigmoid(C_phi(s_i)); set a_i=(epsilon+c_i)^alpha; sample indices with probability q_i proportional to the original pool probability p_i times a_i; compute r_i=p_i/q_i; optionally clip r_i to [0,w_max] and normalize by the batch mean; update theta using the weighted loss sum_i r_i ell_theta(s_i). Refresh scores periodically and reduce alpha if ESS/B falls below a threshold. 3. Computed from the paper: BCE criticality learning, policy-outcome labels, failure-focused replacement, and importance-weighted resampling. Estimated empirically: calibration of C_phi, q_i, normalization, and the effect of clipping. 4. First cheap experiment: train an actor-critic or behavior-cloning MLP on CartPole with injected initial-state and action perturbations. Compare uniform replay against alpha in {0.5,1,2}, with and without importance weighting. The predicted signatures are monotonic growth of sampled failure fraction with alpha, decreasing ESS/B, and better agreement between weighted-batch and large-uniform-batch gradients than between unweighted-biased and uniform gradients. The measured enrichment ratio should match the q_alpha prediction within 20%; use the largest alpha with ESS/B above 0.3 and weighted-gradient cosine similarity above 0.9.