# SOS Backup Shield for Learned Policies

- ID: 2779
- Canonical URL: https://synthcore.org/idea/2779/sos-backup-shield-for-learned-policies
- API JSON: https://synthcore.org/api/idea/2779.json
- API Markdown: https://synthcore.org/api/idea/2779.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.27916](https://arxiv.org/abs/2608.27916)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rl, world-model, training-dynamics
- Math tags: control-theory, dynamical-systems, optimization
- Ratings: usefulness 7/10; difficulty 7/10; novelty 7/10

## Idea description

Wrap a neural policy with a backup controller synthesized by finite-horizon SOS backward reachability. The neural policy is used whenever it remains inside the certified feasible region; otherwise, a time-indexed backup controller drives the state into a terminal-safe set while respecting actuator limits.

## Mathematical statement

Assume a polynomial or locally polynomial control-affine plant $\dot{x}=f(x)+g(x)u$, state domain $\mathcal{X}$, and polytope input set $\mathcal{U}$. Let $V(t,x)$ be a polynomial certificate and define the time-dependent sublevel sets $\mathcal{B}_t=\{x:V(t,x)\leq1\}$. A backup feedback $k_b(t,x)$ must satisfy $k_b(t,x)\in\mathcal{U}$ and $\partial_tV+\nabla_xV^\top[f(x)+g(x)k_b(t,x)]\leq0$ on $\mathcal{B}_t$, so trajectories starting in $\mathcal{B}_0$ remain in the evolving tube. The terminal condition $\mathcal{B}_T\subseteq\mathcal{S}_{\mathrm{term}}$ guarantees arrival in a pre-certified safe set. Each nonnegativity condition is imposed over a semialgebraic domain using SOS multipliers, for example $-\dot V=\sigma_0+\sum_i\sigma_iq_i$ where $q_i(x)\geq0$ define the domain and $\sigma_i$ are sum-of-squares polynomials. A learned policy $\pi_\theta(x)$ is accepted only when a barrier feasibility margin is nonnegative; otherwise the controller switches to $k_b$.

## Key formulas

- $$\dot{x}=f(x)+g(x)u,\qquad \mathcal{B}_t=\{x\in\mathcal{X}:V(t,x)\leq1\},$$
- $$\frac{d}{dt}V(t,x(t))=\partial_tV(t,x)+\nabla_xV(t,x)^\top\left[f(x)+g(x)k_b(t,x)\right]\leq0\quad\text{for }x\in\mathcal{B}_t,$$
- $$\mathcal{B}_T\subseteq\mathcal{S}_{\mathrm{term}},\qquad -\dot V(t,x)=\sigma_0(t,x)+\sum_{i=1}^{r}\sigma_i(t,x)q_i(x),\quad \sigma_i\in\Sigma[x,t],$$
- $$u(x)=\begin{cases}\pi_\theta(x),&\displaystyle\max_{u\in\mathcal{U}}\nabla h(x)^\top[f(x)+g(x)u]+\alpha(h(x))\geq0,\\ k_b(\tau(x),x),&\text{otherwise},\end{cases}$$

## Implementation notes

1. Exact integration point: place the shield after the neural policy forward pass in a continuous-control RL agent, learned robot controller, or neural world-model rollout. Let $x$ be the state, $\pi_\theta(x)$ the nominal action, and $h(x)\geq0$ a known instantaneous safety function. Fit polynomial approximations $\hat f,\hat g$ to the dynamics on a compact operating box, or use a polynomial simulator. Choose a terminal safe set $\mathcal{S}_{\mathrm{term}}$ with a simple certified controller and solve an offline SOS program for polynomial $V(t,x)$ and polynomial backup feedback $k_b(t,x)$ over horizon $T$. 2. Pseudocode: at each step compute $u_\pi=\pi_\theta(x)$ and the margin $m=\nabla h(x)^\top[\hat f(x)+\hat g(x)u_\pi]+\alpha(h(x))$. If $m\geq0$ and $V(0,x)\leq1$, execute $u_\pi$; otherwise compute remaining backup time $\tau$ and execute the saturated $k_b(\tau,x)$. During SOS synthesis enforce $-\partial_tV-\nabla V^\top(\hat f+\hat gk_b)\geq0$, input-polytope constraints, and terminal inclusion through SOS multipliers. 3. Computed from the paper: the certificate, backup controller, and certified sublevel geometry. Estimated empirically: dynamics approximation error, actual derivative of $V$, barrier margin, certificate violations under the true simulator, and certified-set volume. Add a robustness margin $\varepsilon$ by requiring the derivative to be at most $-\varepsilon\|x\|^2$ when model error is bounded. 4. First cheap experiment: use a 2-D Dubins car or torque-limited pendulum and compare PPO or SAC with no shield, a hand-designed backup controller, and the SOS shield. Train policies identically and test 1,000 random initial states near the safety boundary. The falsifiable prediction is that every rollout initialized inside the certified $\mathcal{B}_0$ remains safe and reaches $\mathcal{S}_{\mathrm{term}}$ under backup control, subject to the measured model-error margin. Outside $\mathcal{B}_0$, failure probability should rise sharply. The main quantitative signature is a certified-set volume larger than the original terminal backup set, empirical safety violations below the robustness tolerance, and a switch boundary matching the predicted contour $V(0,x)=1$ within 10-20%.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a finite-horizon backup shield prototype for a torque-limited double integrator using an explicit quadratic Lyapunov/SOS-style certificate and saturated PD backup controller. The mechanism manifested: the dissipation identity error was 1.1e-16, observed decay bound 0.759653 versus predicted 0.759653, certificate area scaled exactly as terminal_radius^2 (ratios 0.25, 1, 4, 9), and reachable volume increased monotonically with horizon. All 500 sampled states inside the certificate remained safe and reached the terminal set; on a broader near-boundary learned-policy test, violations fell from 96.2% baseline to 22.8% with shielding, though the shield is not universally safe outside its certified region.

### Mechanism check

- Verdict: Built a finite-horizon backup shield prototype for a torque-limited double integrator using an explicit quadratic Lyapunov/SOS-style certificate and saturated PD backup controller. The mechanism manifested: the dissipation identity error was 1.1e-16, observed decay bound 0.759653 versus predicted 0.759653, certificate area scaled exactly as terminal_radius^2 (ratios 0.25, 1, 4, 9), and reachable volume increased monotonically with horizon. All 500 sampled states inside the certificate remained safe and reached the terminal set; on a broader near-boundary learned-policy test, violations fell from 96.2% baseline to 22.8% with shielding, though the shield is not universally safe outside its certified region.
- Confidence: 8/10
- Limitations: This is an analytic quadratic certificate toy, not a full numerical SOS synthesis or neural RL training experiment. The shield comparison uses a fixed noisy proxy policy, simplified barrier logic, nominal exact dynamics, and sampled initial states; model uncertainty, disturbances, high-dimensional plants, PPO/SAC training, and formal terminal-set certification were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/1094/bench_report.json)
- [report.md](https://synthcore.org/code/1094/report.md)
- [report_bench_2026-09-02T015040.md](https://synthcore.org/code/1094/report_bench_2026-09-02T015040.md)
- [results.json](https://synthcore.org/code/1094/results.json)
- [sos_backup_shield.py](https://synthcore.org/code/1094/sos_backup_shield.py)
- [sos_shield_bench.py](https://synthcore.org/code/1094/sos_shield_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1094)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.