# Composed Hamilton-Jacobi Reachability Critics

- ID: 2492
- Canonical URL: https://synthcore.org/idea/2492/composed-hamilton-jacobi-reachability-critics
- API JSON: https://synthcore.org/api/idea/2492.json
- API Markdown: https://synthcore.org/api/idea/2492.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.18060](https://arxiv.org/abs/2608.18060)
- Category: dynamics
- Solves: stability, accuracy, sample-efficiency
- ML areas: rl, world-model, training-dynamics, loss
- Math tags: control-theory, pde, dynamical-systems, optimization
- Ratings: usefulness 8/10; difficulty 6/10; novelty 7/10

## Idea description

Attach one neural value head to each generalized reach-avoid subtask and compose these heads into a critic for sequential or timed temporal-logic goals. The policy is trained to increase the composed value while an auxiliary Hamilton-Jacobi residual trains each local head against the learned or known dynamics. This replaces a single poorly conditioned long-horizon objective with short-horizon certificates whose composition has an explicit logical meaning.

## Mathematical statement

Let $x\in\mathbb{R}^{n}$ be the state, $u\in\mathcal{U}$ the policy control, $d\in\mathcal{D}$ an adversarial disturbance, and $\dot{x}=f(x,u,d,t)$ the dynamics. For subtask $i$, let $G_i(x,t)$ be a signed target function, positive inside the target, and $A_i(x,t)$ a signed avoid function, positive inside the forbidden set. A differentiable GRA value $V_i(x,t)$ is trained to satisfy the robust Hamilton-Jacobi equation $\partial_tV_i+\min_{u\in\mathcal U}\max_{d\in\mathcal D}\nabla_xV_i^\top f(x,u,d,t)=0$ away from target and avoid boundaries, with obstacle boundary conditions encoding target attainment and avoidance. For a sequential specification requiring subtask 1 then subtask 2, use the value composition $V_{1\rightarrow2}(x,t)=\sup_{\tau\in[t,T]}\min\{V_1(x,t;\tau),V_2(x,\tau)\}$, where $\tau$ is the switching time; for conjunction use $\min_iV_i$, and for disjunction use $\max_iV_i$. The robust policy action is selected by minimizing the local Hamiltonian, $u^*(x,t)=\arg\min_u\max_d\nabla_xV_{\mathrm{comp}}^\top f(x,u,d,t)$. The useful property is that local PDE residuals and explicit max/min compositions expose which subtask causes failure instead of hiding it in one scalar return.

## Key formulas

- $$\dot{x}(t)=f(x(t),u(t),d(t),t),\qquad \partial_tV_i(x,t)+\min_{u\in\mathcal U}\max_{d\in\mathcal D}\nabla_xV_i(x,t)^\top f(x,u,d,t)=0.$$
- $$\mathcal{R}_i(x,t)=\partial_tV_i(x,t)+\min_{u\in\mathcal U}\max_{d\in\mathcal D}\nabla_xV_i(x,t)^\top f_\theta(x,u,d,t),\qquad \mathcal L_{\mathrm{HJ}}=\sum_i\mathbb E[|\mathcal R_i|^2].$$
- $$V_{1\rightarrow2}(x,t)=\sup_{\tau\in[t,T]}\min\{V_1(x,t;\tau),V_2(x,\tau)\},\qquad V_{\wedge}(x,t)=\min_iV_i(x,t),\quad V_{\vee}(x,t)=\max_iV_i(x,t).$$
- $$u^*(x,t)=\arg\min_{u\in\mathcal U}\max_{d\in\mathcal D}\nabla_xV_{\mathrm{comp}}(x,t)^\top f_\theta(x,u,d,t).$$

## Implementation notes

1. Exact integration point: use a model-based RL or neural controlled-dynamics setup. Let $f_\theta$ be either a known simulator, a learned differentiable dynamics model, or a neural ODE. Add $K$ MLP heads $V_{\phi_i}(x,t)$, one for each target/avoid primitive. Keep the policy $\pi_\psi(u|x,t)$ separate; during actor updates, maximize the composed value at the initial state. 2. Pseudocode: sample states $x,t$, controls $u$, and disturbances $d$; evaluate $f_\theta$ and automatic derivatives of every $V_{\phi_i}$; approximate the min/max Hamiltonian with softmin/softmax temperature $\beta$; compute $\mathcal R_i=\partial_tV_i+\operatorname{softmin}_u\operatorname{softmax}_d(\nabla_xV_i^\top f_\theta)$; apply target and avoid obstacle losses; form $V_{\mathrm{comp}}$ with soft versions of the stated max/min and a switching-time grid; update critic parameters using $\mathcal L=\mathcal L_{\mathrm{HJ}}+\lambda_b\mathcal L_{\mathrm{boundary}}+\lambda_c\mathcal L_{\mathrm{composition}}$, then update the actor toward $u^*$. 3. Computed from the paper: the GRA decomposition and max/min temporal composition. Estimated empirically: $f_\theta$, PDE derivatives, disturbance maximization, and boundary signs. 4. First cheap experiment: use a 2-D Dubins or point-mass system with bounded control and disturbance, and a timed goal sequence such as reach region A by time 1, then region B by time 2 while avoiding a disk. Compare SAC or model-predictive control against one monolithic value head and the composed critic. Measure success, minimum obstacle margin, HJ residual, and calibration of the sign of $V_{\mathrm{comp}}$. The quantitative prediction is that the composed critic's success boundary in initial-state space should agree with a gridded dynamic-programming reachability solution within 10% Hausdorff distance, while halving the temporal subtask horizon should reduce one-step composition error approximately linearly with the integration step. A second signature is a sharp empirical transition: initial states with $V_{\mathrm{comp}}>0$ should have substantially higher success probability than states with $V_{\mathrm{comp}}<0$; tune the threshold on a validation grid and report the calibration curve rather than only average return.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a self-contained composed reachability critic toy for bounded 1D dynamics, including local signed value heads, sequential max-min composition, logical min/max composition, HJ residual checking, switch-time discretization, soft composition, and a monolithic neural comparison. The mechanism manifested: the affine HJ residual was 2.7e-11, logical sign composition had zero violations, switch-grid error scaled linearly as 0.48 times the time step, and softmin error followed the predicted ln(2)/beta bound. The composed analytic critic achieved 1.000 test sign accuracy versus 0.981 for the small monolithic neural head, although this is not a full RL study.

### Mechanism check

- Verdict: Built a self-contained composed reachability critic toy for bounded 1D dynamics, including local signed value heads, sequential max-min composition, logical min/max composition, HJ residual checking, switch-time discretization, soft composition, and a monolithic neural comparison. The mechanism manifested: the affine HJ residual was 2.7e-11, logical sign composition had zero violations, switch-grid error scaled linearly as 0.48 times the time step, and softmin error followed the predicted ln(2)/beta bound. The composed analytic critic achieved 1.000 test sign accuracy versus 0.981 for the small monolithic neural head, although this is not a full RL study.
- Confidence: 8/10
- Limitations: This is a low-dimensional analytic toy, not a learned multi-head critic trained with HJ losses, actor updates, or adversarial disturbances in an RL loop. It does not test Dubins or point-mass dynamics, obstacle-boundary training, calibration curves, Hausdorff reachability error, sample efficiency, FLOPs, or long-horizon policy success.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/849/bench_report.json)
- [experiment.py](https://synthcore.org/code/849/experiment.py)
- [report.md](https://synthcore.org/code/849/report.md)
- [report_bench_2026-09-01T123027.md](https://synthcore.org/code/849/report_bench_2026-09-01T123027.md)
- [results.json](https://synthcore.org/code/849/results.json)
- [stage2_bench.py](https://synthcore.org/code/849/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/849)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.