Reachability Trust Region for Policy Updates

1. Integration point: modify PPO, SAC, or supervised policy-gradient updates with a reachability-based line search. Every K optimizer steps, roll out the current policy through a differentiable local dynamics model for H steps and propagate zonotopes under the current disturbance estimate. 2. Pseudocode: compute the ordinary gradient g; set a candidate direction d = -g; evaluate the nominal loss and reachable tube for theta plus alpha d; start alpha at 1 and repeatedly halve alpha until all predicted tube points satisfy safety constraints and the tube-growth condition s_H(theta plus alpha d) - s_H(theta) <= tau. Apply the accepted update, then collect data and refine W from transition residuals. A cheaper version uses the first-order estimate s_hat = s_H + alpha times grad_s transposed times d before exact checking. 3. Computed from the paper mechanism: reachable-set propagation and safety-set containment. Estimated empirically: A_k, B_k, policy Jacobians, the horizon H, and the tube budget tau. Use conservative interval bounds for neural-network action Jacobians; if unavailable, estimate the tube under several perturbation samples and label the method heuristic rather than formally certified. 4. First cheap experiment: use a two-dimensional nonlinear point mass or cart-pole with bounded external force and compare PPO with KL clipping against PPO plus the reachability line search. Sweep tau and H. The predicted signature is a sharp transition in accepted optimizer step size: once the proposed update causes safety-boundary contact or tube growth greater than tau, backtracking should reduce alpha, and constraint violations should remain near zero below the model-error coverage level. Report the empirical largest alpha satisfying the tube test and check whether violations rise abruptly when alpha is increased beyond that boundary.