# Reachability Trust Region for Policy Updates

- ID: 2754
- Canonical URL: https://synthcore.org/idea/2754/reachability-trust-region-for-policy-updates
- API JSON: https://synthcore.org/api/idea/2754.json
- API Markdown: https://synthcore.org/api/idea/2754.md
- Verification status: unverified
- Source: [arXiv:2608.26852](https://arxiv.org/abs/2608.26852)
- Category: optimization
- Solves: stability, accuracy
- ML areas: rl, optimizer, training-dynamics
- Math tags: control-theory, dynamical-systems, linear-algebra, geometry
- Ratings: usefulness 6/10; difficulty 7/10; novelty 8/10

## Idea description

Use the change in the policy-induced reachable set as a trust-region constraint, rather than limiting only parameter distance or KL divergence. A policy update is accepted when its predicted finite-horizon zonotope remains sufficiently close to the previous reachable tube and does not cross the safety boundary, yielding a dynamics-aware step-size ceiling.

## Mathematical statement

Let pi_theta be a neural policy and let R_H(theta) denote a zonotopic over-approximation of states reachable for H steps under pi_theta, starting from X_0 and disturbance zonotope W. Define a scalar tube radius s_H(theta), for example the sum of generator norms across the horizon, s_H(theta) = sum from k=1 to H of ||G_k(theta)||_1. A candidate update theta_prime is accepted if R_H(theta_prime) is a subset of the safe set S and s_H(theta_prime) - s_H(theta) is at most tau, where tau is a prescribed tube-growth budget. For a differentiable surrogate, linearize s_H in the update Delta theta and impose grad_theta s_H transposed times Delta theta <= tau, or backtrack the optimizer step until the full zonotope test passes. This transfers the paper's online reachability test into a policy-optimization stability certificate.

## Key formulas

- R_(k+1)(theta) = A_k R_k(theta) plus B_k pi_theta(R_k(theta)) plus d_k plus W_k,   R_0(theta) = X_0
- s_H(theta) = sum from k=1 to H of ||G_k(theta)||_1,   R_H(theta_prime) is a subset of S,   s_H(theta_prime) - s_H(theta) <= tau
- grad_theta s_H(theta) transposed times Delta theta <= tau,   theta_prime = theta + alpha Delta theta

## Implementation notes

1. Integration point: modify PPO, SAC, or supervised policy-gradient updates with a reachability-based line search. Every K optimizer steps, roll out the current policy through a differentiable local dynamics model for H steps and propagate zonotopes under the current disturbance estimate. 2. Pseudocode: compute the ordinary gradient g; set a candidate direction d = -g; evaluate the nominal loss and reachable tube for theta plus alpha d; start alpha at 1 and repeatedly halve alpha until all predicted tube points satisfy safety constraints and the tube-growth condition s_H(theta plus alpha d) - s_H(theta) <= tau. Apply the accepted update, then collect data and refine W from transition residuals. A cheaper version uses the first-order estimate s_hat = s_H + alpha times grad_s transposed times d before exact checking. 3. Computed from the paper mechanism: reachable-set propagation and safety-set containment. Estimated empirically: A_k, B_k, policy Jacobians, the horizon H, and the tube budget tau. Use conservative interval bounds for neural-network action Jacobians; if unavailable, estimate the tube under several perturbation samples and label the method heuristic rather than formally certified. 4. First cheap experiment: use a two-dimensional nonlinear point mass or cart-pole with bounded external force and compare PPO with KL clipping against PPO plus the reachability line search. Sweep tau and H. The predicted signature is a sharp transition in accepted optimizer step size: once the proposed update causes safety-boundary contact or tube growth greater than tau, backtracking should reduce alpha, and constraint violations should remain near zero below the model-error coverage level. Report the empirical largest alpha satisfying the tube test and check whether violations rise abruptly when alpha is increased beyond that boundary.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.