# Monotone Compositional Reachability Critic

- ID: 2763
- Canonical URL: https://synthcore.org/idea/2763/monotone-compositional-reachability-critic
- API JSON: https://synthcore.org/api/idea/2763.json
- API Markdown: https://synthcore.org/api/idea/2763.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.27654](https://arxiv.org/abs/2608.27654)
- Category: architecture
- Solves: stability, accuracy, scalability
- ML areas: rl, world-model, loss, regularization
- Math tags: control-theory, dynamical-systems, pde, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Train separate neural value functions for primitive reachability, avoidance, or target-reaching tasks, then combine them with a coordinatewise monotone aggregator whose derivatives with respect to all primitive values are nonnegative. This transfers the paper's exact two-player decomposition condition into a modular critic architecture: adding a new target changes only one primitive critic and the aggregator, rather than requiring a new high-dimensional value function.

## Mathematical statement

Consider continuous-time dynamics \(\dot{x}=f(x,u,d,t)\) with state \(x\in\mathbb{R}^n\), control \(u\in\mathcal U\), disturbance \(d\in\mathcal D\), and finite horizon \(t\in[0,T]\). For a value function \(V(x,t)\), the two-player Hamilton-Jacobi equation is \(\partial_tV+H(x,\nabla_xV,t)=0\), with Hamiltonian \(H(x,p,t)=\min_{u\in\mathcal U}\max_{d\in\mathcal D}p^\top f(x,u,d,t)\). Let \(V_i(x,t)\) denote primitive value functions and let the composite task value be \(V(x,T)=\Phi(V_1(x,T),\ldots,V_m(x,T))\). The transferable condition is coordinatewise monotonicity: \(\partial\Phi/\partial z_i\ge0\) for every primitive coordinate \(z_i\). Implement \(\widehat V(x,t)=\Phi_\theta(\widehat V_1(x,t),\ldots,\widehat V_m(x,t))\) with nonnegative aggregator derivatives. The property to test is preservation of the composite value and its zero-sublevel reachable set; if monotonicity is violated, the ordering of primitive task values can be reversed and the decomposition can fail.

## Key formulas

- $$\dot{x}(t)=f(x(t),u(t),d(t),t),\qquad x\in\mathbb R^n,\ u\in\mathcal U,\ d\in\mathcal D.$$
- $$\partial_t V(x,t)+\min_{u\in\mathcal U}\max_{d\in\mathcal D}\nabla_xV(x,t)^\top f(x,u,d,t)=0.$$
- $$\widehat V(x,t)=\Phi_\theta(\widehat V_1(x,t),\ldots,\widehat V_m(x,t)),\qquad \frac{\partial\Phi_\theta}{\partial z_i}(z)\ge0\quad\forall i,z.$$
- $$\mathcal R_t=\{x:V(x,t)\le0\},\qquad \widehat{\mathcal R}_t=\{x:\widehat V(x,t)\le0\};\quad \text{monotone decomposition predicts }\widehat{\mathcal R}_t\approx\mathcal R_t.$$

## Implementation notes

1. Exact integration point: use this as a critic for continuous-control RL, model-predictive safety filtering, or neural HJ reachability. Train one network \(V_i(x,t)\) per primitive target or constraint and a small aggregator network \(\Phi_\theta\) on their scalar outputs. Use a monotone lattice or positive-weight network, for example \(\Phi_\theta(z)=b+\sum_k a_k\,\mathrm{softplus}(w_k^\top z+c_k)\), with \(a_k\ge0\) and \(w_{ki}\ge0\) enforced by softplus parameterizations. This guarantees nonnegative partial derivatives. 2. Pseudocode: sample \((x,t)\), controls, and disturbances; compute \(z_i=V_i(x,t)\); set \(V=\Phi_\theta(z)\); obtain \(\nabla_xV\) by automatic differentiation; approximate the Hamiltonian using sampled controls and disturbances or differentiable soft min/max; minimize \(|\partial_tV+H(x,\nabla_xV,t)|^2\) plus terminal error \(|V(x,T)-\Phi_\theta(V_i(x,T))|^2\). Pretrain each primitive critic using a low-dimensional HJ solver or its own PDE residual. 3. Quantities computed directly are network gradients, PDE residuals, aggregator derivatives, and zero-sublevel masks. The theorem supplies the monotonicity condition; decomposition error, Hamiltonian error, and set mismatch must be estimated empirically. Use adversarial rollouts with approximate maximizing disturbances to test the game value. 4. First cheap experiment: use a two-dimensional Dubins or planar single-integrator system with bounded control and disturbance, two target disks, and one obstacle. Compare a monolithic value network, an unconstrained MLP aggregator, and the positive-weight monotone aggregator against a grid-based finite-horizon HJ solution. Measure value RMSE, PDE residual, reachable-set symmetric difference, and adversarial rollout violations. The quantitative prediction is that the monotone model's set error remains near the propagated primitive errors, while the unconstrained model fails when \(\partial\Phi/\partial z_i<0\). Sweep one aggregator coefficient through zero; reachable-set mismatch and violation rate should increase sharply after a negative-derivative region appears. Target a monotone-model set IoU above 0.9 and an unconstrained residual at least 2 times larger after the sign violation.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a monotone positive-weight compositional critic and numerically tested its core claims. The aggregator preserved coordinatewise ordering in 100% of trials, primitive-value error propagated linearly with log-log exponent 0.9999, and the direct order-reversal transition occurred exactly when the swept coefficient became negative (0% reversal for c2>=0, 100% for c2<0). In the matched synthetic critic task, monotone and unconstrained aggregators performed essentially identically, so this verifies the mechanism but does not demonstrate a practical accuracy win.

### Mechanism check

- Verdict: Built a monotone positive-weight compositional critic and numerically tested its core claims. The aggregator preserved coordinatewise ordering in 100% of trials, primitive-value error propagated linearly with log-log exponent 0.9999, and the direct order-reversal transition occurred exactly when the swept coefficient became negative (0% reversal for c2>=0, 100% for c2<0). In the matched synthetic critic task, monotone and unconstrained aggregators performed essentially identically, so this verifies the mechanism but does not demonstrate a practical accuracy win.
- Confidence: 8/10
- Limitations: This is a small synthetic static value-function experiment, not a continuous-time HJ solver, adversarial rollout, RL critic, or learned nonlinear aggregator. The unconstrained model was trained on data generated by a monotone target, so it learned positive weights and had no opportunity to expose a practical generalization advantage. Runtime, scalability, and high-dimensional reachability were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.56675
- Paired wins: 4/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/1091/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/1091/bench_report.json)
- [experiment.py](https://synthcore.org/code/1091/experiment.py)
- [report.md](https://synthcore.org/code/1091/report.md)
- [report_bench_2026-09-02T002101.md](https://synthcore.org/code/1091/report_bench_2026-09-02T002101.md)
- [results.json](https://synthcore.org/code/1091/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1091)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.