# Lyapunov-Certified Policy Training

- ID: 2823
- Canonical URL: https://synthcore.org/idea/2823/lyapunov-certified-policy-training
- API JSON: https://synthcore.org/api/idea/2823.json
- API Markdown: https://synthcore.org/api/idea/2823.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.29303](https://arxiv.org/abs/2608.29303)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rl, training-dynamics, world-model, regularization
- Math tags: control-theory, dynamical-systems, optimization, linear-algebra
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Train a neural policy together with a positive neural Lyapunov function so that the learned closed-loop transition decreases the function at every sampled state in a prescribed operating region. This converts policy learning from an unconstrained reward problem into a constrained dissipativity problem and provides an inference-time monitor that can reject or damp actions when the certificate is violated.

## Mathematical statement

Let $x\in\mathbb{R}^{n}$ be the state, $u=\pi_{\theta}(x)$ the neural controller with parameters $\theta$, and $\hat f_{\phi}(x,u)$ the identified one-step dynamics model with parameters $\phi$. Let $V_{\psi}(x)$ be a neural Lyapunov candidate with parameters $\psi$, equilibrium $x^{\star}$, and region $\mathcal{X}$. Enforce $V_{\psi}(x^{\star})=0$, $V_{\psi}(x)\geq c_{1}\|x-x^{\star}\|_{2}^{2}$, and $V_{\psi}(\hat f_{\phi}(x,\pi_{\theta}(x))) - V_{\psi}(x)\leq -c_{2}\|x-x^{\star}\|_{2}^{2}$ for all $x\in\mathcal{X}$, where $c_{1},c_{2}>0$. The decrease inequality implies geometric decay of the sampled closed-loop energy when the dynamics are discrete-time. In practice, optimize hinge penalties for violations and periodically verify the inequalities over boxes or a sampled state set; the certified subset is the set on which all inequalities hold.

## Key formulas

- $$V_{\psi}(x^{\star})=0,\qquad V_{\psi}(x)\geq c_{1}\|x-x^{\star}\|_{2}^{2},\qquad c_{1}>0,$$
- $$\Delta V_{\theta,\phi,\psi}(x)=V_{\psi}\!\left(\hat f_{\phi}(x,\pi_{\theta}(x))\right)-V_{\psi}(x)\leq -c_{2}\|x-x^{\star}\|_{2}^{2},\qquad c_{2}>0,$$
- $$\mathcal{L}=\mathcal{L}_{\mathrm{task}}+\lambda_{V}\mathbb{E}_{x\sim\mathcal{D}}[\operatorname{softplus}(\varepsilon_{V}-V_{\psi}(x))]+\lambda_{\Delta}\mathbb{E}_{x\sim\mathcal{D}}[\operatorname{softplus}(\Delta V_{\theta,\phi,\psi}(x)+c_{2}\|x-x^{\star}\|_{2}^{2})],$$
- $$V_{t+1}\leq\left(1-\frac{c_{2}}{c_{1}}\right)V_{t}\quad\Longrightarrow\quad V_{t}\leq\left(1-\frac{c_{2}}{c_{1}}\right)^{t}V_{0},\qquad 0<c_{2}<c_{1}.$$

## Implementation notes

1. Integration point: use this in model-based RL, recurrent world-model rollout training, or any neural state-transition module where stability of generated trajectories matters. Maintain three networks: a dynamics model $\hat f_{\phi}$, policy $\pi_{\theta}$, and Lyapunov network $V_{\psi}$. Parameterize $V_{\psi}(x)=\|g_{\psi}(x)-g_{\psi}(x^{\star})\|_{2}^{2}+\epsilon\|x-x^{\star}\|_{2}^{2}$ with $\epsilon>0$ to make positivity automatic. 2. Pseudocode: sample states $x$ from replay data and perturbations around the equilibrium; compute $u=\pi_{\theta}(x)$, $x^{+}=\hat f_{\phi}(x,u)$, and $dV=V_{\psi}(x^{+})-V_{\psi}(x)$; add the two hinge terms in the displayed loss; update $\theta,\psi$ jointly and update $\phi$ using the transition prediction loss. Every $K$ steps, freeze the networks and test $V(x)>0$ and $dV+c_{2}\|x-x^{\star}\|^{2}\leq0$ on a dense grid or interval boxes. 3. Computed quantities are the one-step model prediction, Lyapunov values, and violation margins. Model error and the largest verified region must be estimated from held-out transitions or interval bounds. 4. First cheap experiment: train a two-layer policy and Lyapunov network on a two-dimensional nonlinear benchmark such as $x_{t+1}=x_t+0.05[-x_{1}^{3}+u,\ x_{2}+0.2x_{1}]$, comparing PPO or supervised control against the certificate-regularized version. Sweep $\lambda_{\Delta}$ and estimate the largest radius $r$ for which the decrease condition holds. The quantitative prediction is that certified trajectories satisfy $V_t/V_0\leq(1-c_{2}/c_{1})^t$ up to model error, and empirical divergence begins near the radius or parameter value where the measured minimum decrease margin crosses zero.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a PyTorch MVP with an equilibrium-preserving policy, positive Lyapunov network, hinge certificate loss, grid verification, radius sweep, and rollout monitoring in experiment.py. The algebraic recurrence V_{t+1}<=qV_t was verified synthetically, and increasing certificate weight reduced sampled violation from 0.934 (lambda=0) to 0.00030 (lambda=4), but the claimed contraction factor q=0.6875 was not observed: certified one-step V ratios were roughly 0.90-0.99. This exposes a limitation in the stated math: V>=c1||x||^2 alone cannot convert decrease proportional to ||x||^2 into geometric decay in V; an upper bound V<=c3||x||^2 is also required. Therefore the practical penalty effect appeared, but the authoritative Lyapunov contraction claim was not confirmed.

### Mechanism check

- Verdict: Built a PyTorch MVP with an equilibrium-preserving policy, positive Lyapunov network, hinge certificate loss, grid verification, radius sweep, and rollout monitoring in experiment.py. The algebraic recurrence V_{t+1}<=qV_t was verified synthetically, and increasing certificate weight reduced sampled violation from 0.934 (lambda=0) to 0.00030 (lambda=4), but the claimed contraction factor q=0.6875 was not observed: certified one-step V ratios were roughly 0.90-0.99. This exposes a limitation in the stated math: V>=c1||x||^2 alone cannot convert decrease proportional to ||x||^2 into geometric decay in V; an upper bound V<=c3||x||^2 is also required. Therefore the practical penalty effect appeared, but the authoritative Lyapunov contraction claim was not confirmed.
- Confidence: 9/10
- Limitations: Only a tiny 2D deterministic benchmark was tested; no learned dynamics model, transition-model error, held-out generalization, interval certification, PPO/RL training, action rejection monitor, or multiple random seeds was evaluated. The radius sweep has grid-resolution and near-equilibrium numerical sensitivity.

## Artifacts

- [experiment.py](https://synthcore.org/code/1066/experiment.py)
- [report.md](https://synthcore.org/code/1066/report.md)
- [results.json](https://synthcore.org/code/1066/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1066)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.