Residual-Scenario Safety Training

Integrate this at the output head of either a learned multi-step dynamics model or a neural policy. Let a frozen encoder produce $\phi_t=\phi_\psi(z_t)\in\mathbb{R}^p$, and let the trainable head produce nominal outputs $\hat y_{t:t+N-1}=W\Phi_t+b$, where $W,b$ are the only optimized variables during the first experiment. Maintain a FIFO residual buffer $\mathcal B$ containing observed errors $e_t=y_t-\hat y_t$; normalize residuals componentwise and remove only logging artifacts. At each minibatch, draw $N_{\mathrm{scen}}$ residual trajectories $s^{(i)}=[s_0^{(i)};\ldots;s_{N-1}^{(i)}]$ by bootstrap sampling blocks from $\mathcal B$. Compute $y^{(i)}=\hat y+s^{(i)}$. For box constraints $\ell\leq y\leq u$, compute $h_i=\max(0,\ell-y^{(i)},y^{(i)}-u)$ elementwise and use $L=L_{\mathrm{nominal}}+\lambda_W\|W\|_F^2+\mu\sum_i\|h_i\|_1$. Alternatively solve the convex frozen-head problem directly with projected gradient or a QP solver when the nominal loss is quadratic. Update the buffer online after every prediction window, optionally conditioning sampling on regime labels or nearest-neighbor state regions so scenarios reflect operating-point-dependent errors. The paper supplies the scenario perturbation, all-scenario constraint logic, and exact-penalty mechanism; estimate residuals empirically and estimate the effective head dimension $d$ as the number of free parameters in $W,b$. If using the confidence calculation, choose violation tolerance $\varepsilon$ and confidence failure probability $\beta$, then increase $N_{\mathrm{scen}}$ until the displayed binomial tail is at most $\beta$. First test on a small GRU or MLP predictor for cart-pole or pendulum, comparing ordinary MSE training, Gaussian-noise augmentation, and residual-scenario training with equal forward-pass budgets. Track nominal MSE, worst-quantile rollout error, fraction of trajectories violating output bounds, average slack, and calibration on a held-out disturbance distribution. Success means substantially fewer 95th/99th-percentile constraint violations with only a small nominal-MSE increase; the frozen-encoder experiment should establish whether the scenario mechanism is useful before end-to-end fine-tuning.