Closed-Loop Barrier Safety Layer

1. Integration point: use a neural policy $r_{\mathrm{nn}}=\pi_\theta(o)$ to generate a reference for an existing stabilizing controller $u=K(x,r)$. Construct or identify the complete closed-loop vector field $\widehat F(x,r)=f(x)+g(x)K(x,r)$, rather than filtering the raw actuator command. Place the QP between the neural policy and the low-level controller. Define one barrier function for every state limit, for example $h_i(x)=x_{i,\max}-x_i$ and $h_i(x)=x_i-x_{i,\min}$, and solve all inequalities jointly. 2. Pseudocode: compute $r_{\mathrm{nn}}=\pi_\theta(o)$; evaluate $h_i(x)$, $\nabla h_i(x)$, and $\widehat F(x,r)$; solve $\min_r\|r-r_{\mathrm{nn}}\|_W^2/2$ subject to $\nabla h_i(x)^\mathsf{T}\widehat F(x,r)+\gamma_i h_i(x)\geq\varepsilon_i(x)$ and reference bounds; send $r^\star$ to $K$. If infeasible, add slacks $s_i\geq0$ and minimize $\|r-r_{\mathrm{nn}}\|_W^2+\rho\sum_i s_i^2$, while logging slack violations. 3. Compute from the mechanism: barrier values, gradients, closed-loop model predictions, QP solutions, and chosen $\gamma_i$. Estimate empirically: model-error margins $\varepsilon_i(x)$ from held-out transition residuals, using a high quantile of $|\nabla h_i^\mathsf{T}(F-\widehat F)|$. Differentiable training can use implicit differentiation through the QP; a forward-only filter is sufficient for the first test. 4. First cheap experiment: train PPO or SAC on a 2D point-mass task with position and velocity limits, comparing an unfiltered policy, raw-action CBF filtering, and closed-loop-reference filtering with the same nominal controller. Sweep $\gamma$, $W$, and the robustness margin. The prediction is that the robust filtered system has minimum measured barrier value $h_{\min}\geq-\delta$, where $\delta$ is set by solver tolerance and residual-bound violations. Near the boundary, $\|r^\star-r_{\mathrm{nn}}\|$ should increase sharply; away from it, the correction should be approximately zero and nominal settling behavior should be unchanged. Increasing $\gamma$ should reduce recovery time with an approximate exponential law $h(t)\sim e^{-\gamma t}h(0)$, while increasing correction magnitude and potentially reducing QP feasibility.