# Dual recoverability gate for policy switching

- ID: 3048
- Canonical URL: https://synthcore.org/idea/3048/dual-recoverability-gate-for-policy-switching
- API JSON: https://synthcore.org/api/idea/3048.json
- API Markdown: https://synthcore.org/api/idea/3048.md
- Verification status: mechanism_failed
- Source: [arXiv:2609.02358](https://arxiv.org/abs/2609.02358)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rl, world-model, moe-routing
- Math tags: control-theory, dynamical-systems, probability
- Ratings: usefulness 7/10; difficulty 6/10; novelty 6/10

## Idea description

Attach two independent critics to a learned controller: an empirical probability that executing the controller reaches a desired terminal set, and a recoverability value estimating whether the current state remains inside a safe reach-avoid region. Execute the learned controller only when both critics pass their thresholds; otherwise route to a conservative fallback policy or expert.

## Mathematical statement

The paper defines a safe stopped-state set \(\mathcal{G}_{\mathrm{stop}}\subset\mathcal{S}\), a failure set \(\mathcal{F}\subset\mathcal{S}\), a stop-probability estimator \(\hat P_{\phi}(s)\), and a reach-avoid estimator \(\hat V_{\psi}(s)\). The first predicts the probability that the primary policy reaches \(\mathcal{G}_{\mathrm{stop}}\) rather than \(\mathcal{F}\); the second estimates physical recoverability, with lower values safer under the paper's threshold convention. Here \(s\) is the current state, \(\phi\) and \(\psi\) are critic parameters, and \(\eta_p,\eta_r\) are thresholds. The complementary critics are the transferable asset: \(\hat P_\phi\) captures the actual controller's behavior, while \(\hat V_\psi\) supplies a controller-independent feasibility signal. For a general policy, define \(y=1\) if a rollout reaches goal set \(\mathcal G\) without entering failure set \(\mathcal F\), train \(\hat P_\phi(s)\approx\Pr(y=1\mid s,\pi)\), and train \(\hat V_\psi\) using conservative model or fallback-policy reach-avoid backups. The deployment gate is their conjunction.

## Key formulas

- $$\mathcal{G}_{\mathrm{stop}}\subset\mathcal{S},\qquad \mathcal{F}\subset\mathcal{S}$$
- $$\pi_{\mathrm{SS}}(s_t)=\pi_{\mathrm{stop}}(s_t)\quad\mathrm{iff}\quad \hat P_{\phi}(s_t)\geq\eta_p\ \text{and}\ \hat V_{\psi}(s_t)\leq\eta_r$$
- $$g(s)=\mathbf{1}\!\left[\hat P_{\phi}(s)\geq\eta_p\ \land\ \hat V_{\psi}(s)\leq\eta_r\right],\qquad a_t=g(s_t)\pi(s_t)+(1-g(s_t))\pi_{\mathrm{fallback}}(s_t)$$
- $$\mathcal{L}_{P}=-y\log \hat P_{\phi}(s)-(1-y)\log(1-\hat P_{\phi}(s))$$

## Implementation notes

(1) Integration point: place the gate immediately before action execution in an RL policy, diffusion-action sampler, or MoE router. The primary policy produces the proposed action, while a conservative fallback produces an alternative. Feed both critics the same normalized state features, including velocity, recent actions, proprioception, contact indicators, and task context if relevant. (2) Pseudocode: sample states from several upstream policies; from each state execute the primary policy for a short horizon; set \(y=1\) iff the rollout reaches \(\mathcal G\) without entering \(\mathcal F\); update \(\phi\) with the displayed binary cross-entropy. Separately roll out a fallback policy or learned dynamics model and construct a conservative reach-avoid target for \(\psi\). At inference compute \(p=\hat P_\phi(s)\) and \(v=\hat V_\psi(s)\); execute the primary action iff \(p\geq\eta_p\) and \(v\leq\eta_r\), otherwise execute fallback. (3) The Boolean gate and threshold logic come directly from the paper; probabilities, recoverability targets, and thresholds are estimated. Calibrate \(p\) on held-out trajectories and select thresholds to meet a target failure rate while maximizing primary-policy coverage. (4) First experiment: use PPO on MuJoCo Hopper or Walker2d as the primary controller and a damping or robust-policy controller as fallback. Evaluate under mass perturbations, action delays, observation noise, and initial-state shifts. Compare no gate, one critic, and the dual gate. Success is lower catastrophic-failure rate at equal reward, higher reward at a fixed failure rate, or higher safe primary-policy coverage with similar critic overhead.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a reproducible toy dual-recoverability policy gate with learned primary-success and fallback-failure critics, plus ungated and probability-only baselines. The formula check passed exactly and stricter thresholds produced a subset, while probability gating reduced catastrophic failure from 65.80% to 13.28%; however, the dual gate was identical to the probability-only gate (13.28% failure, 81.16% success, 41.44% primary coverage), so the independent recoverability critic showed no incremental effect in this experiment.

### Mechanism check

- Verdict: Built a reproducible toy dual-recoverability policy gate with learned primary-success and fallback-failure critics, plus ungated and probability-only baselines. The formula check passed exactly and stricter thresholds produced a subset, while probability gating reduced catastrophic failure from 65.80% to 13.28%; however, the dual gate was identical to the probability-only gate (13.28% failure, 81.16% success, 41.44% primary coverage), so the independent recoverability critic showed no incremental effect in this experiment.
- Confidence: 9/10
- Limitations: This is a one-dimensional synthetic control task, not PPO/MuJoCo, and uses logistic critics with hand-designed policies and fixed thresholds. It does not test calibration, delayed observations, multiple disturbances, learned dynamics, compute overhead, or whether a better independent recoverability estimator would add value.

## Artifacts

- [dual_gate_experiment.py](https://synthcore.org/code/1225/dual_gate_experiment.py)
- [report.md](https://synthcore.org/code/1225/report.md)
- [results.json](https://synthcore.org/code/1225/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1225)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.