Lyapunov-Certified Policy Training

1. Integration point: use this in model-based RL, recurrent world-model rollout training, or any neural state-transition module where stability of generated trajectories matters. Maintain three networks: a dynamics model $\hat f_{\phi}$, policy $\pi_{\theta}$, and Lyapunov network $V_{\psi}$. Parameterize $V_{\psi}(x)=\|g_{\psi}(x)-g_{\psi}(x^{\star})\|_{2}^{2}+\epsilon\|x-x^{\star}\|_{2}^{2}$ with $\epsilon>0$ to make positivity automatic. 2. Pseudocode: sample states $x$ from replay data and perturbations around the equilibrium; compute $u=\pi_{\theta}(x)$, $x^{+}=\hat f_{\phi}(x,u)$, and $dV=V_{\psi}(x^{+})-V_{\psi}(x)$; add the two hinge terms in the displayed loss; update $\theta,\psi$ jointly and update $\phi$ using the transition prediction loss. Every $K$ steps, freeze the networks and test $V(x)>0$ and $dV+c_{2}\|x-x^{\star}\|^{2}\leq0$ on a dense grid or interval boxes. 3. Computed quantities are the one-step model prediction, Lyapunov values, and violation margins. Model error and the largest verified region must be estimated from held-out transitions or interval bounds. 4. First cheap experiment: train a two-layer policy and Lyapunov network on a two-dimensional nonlinear benchmark such as $x_{t+1}=x_t+0.05[-x_{1}^{3}+u,\ x_{2}+0.2x_{1}]$, comparing PPO or supervised control against the certificate-regularized version. Sweep $\lambda_{\Delta}$ and estimate the largest radius $r$ for which the decrease condition holds. The quantitative prediction is that certified trajectories satisfy $V_t/V_0\leq(1-c_{2}/c_{1})^t$ up to model error, and empirical divergence begins near the radius or parameter value where the measured minimum decrease margin crosses zero.