△ Mechanism confirmed, baseline not beaten 2026

Feasible Action Mapping Safety Layer

Implementation & benchmark of arXiv:2607.23930 — Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping

Usefulness7/10

Difficulty6/10

Novelty5/10

Architecture Stability Accuracy Sample efficiency Rl Optimizer Training dynamics Control theory Optimization Dynamical systems Geometry

Source paper: Bridging Reinforcement Learning and Optimal Control via Feasible Action Mapping arXiv:2607.23930 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Let a neural policy emit an unconstrained abstract action z, then solve a state-dependent feasibility problem that maps z to an admissible optimal-control parameter p before execution. Unlike coordinate-wise clipping, the mapping accounts for predicted dynamics, coupled state and input constraints, and recursive feasibility, allowing the policy to retain a simple unconstrained output space while the controller enforces plant constraints.

Formulas

$$x_{k+1}=f(x_k,u_k),\qquad u_k=\Phi_k(x_t,p),\qquad k=0,\ldots,H-1,$$

$$\mathcal{Z}_{f}(x_t)=\left\{z:\exists p,\ x_0=x_t,\ x_{k+1}=f(x_k,\Phi_k(x_t,p)),\ x_k\in\mathcal{X},\ \Phi_k(x_t,p)\in\mathcal{U},\ x_H\in\mathcal{X}_f\right\},$$

$$p_t^*=\arg\min_{p}\frac{1}{2}\|p-z_t\|_{W}^{2}\quad\text{s.t.}\quad x_{k+1}=f(x_k,\Phi_k(x_t,p)),\ x_k\in\mathcal{X},\ \Phi_k(x_t,p)\in\mathcal{U},\ x_H\in\mathcal{X}_f,$$

$$a_t=\Phi_0(x_t,p_t^*),\qquad d_t=\|p_t^*-z_t\|_{W};\quad z_t\in\mathcal{Z}_f(x_t)\Rightarrow d_t=0.$$

Mathematical statement

Let x_t be the observed state, z_t = pi_theta(x_t) in R^d the neural policy output, and p be a control parameter or finite-horizon sequence. Let x_{k+1} = f(x_k,u_k), with u_k = Phi_k(x_t,p), horizon H, state constraint x_k in X, input constraint u_k in U, and terminal safe set X_f. The feasible abstract-action set consists of outputs for which some mapped parameter produces a constraint-satisfying trajectory ending in X_f. The safety layer computes the weighted projection p_t^* minimizing the distance from z_t subject to the OCP constraints, then executes Phi_0(x_t,p_t^*). If the model is exact, the optimization remains feasible, and X_f is control invariant, recursive feasibility implies constraint satisfaction at future steps. The measurable signature is zero violations in the certified regime and a sharp increase in projection distance as policy outputs cross the feasible-set boundary.

Implementation notes

1. Integration point: place the layer after the policy network and before the environment or actuator. Begin with a low-dimensional control parameter p, such as constant acceleration, spline coefficients, or an H-step action sequence. The policy outputs z_t, the safety layer solves the constrained finite-horizon problem, and only the first action is sent to the simulator. Use OSQP for linear or quadratic models and IPOPT or CasADi for nonlinear models. Initially stop gradients through the optimizer; later use KKT sensitivity or a differentiable QP layer. 2. Pseudocode: compute z = policy(x); initialize p = z; set x[0] = x; for k in 0..H-1 compute u[k] = Phi_k(x,p), predict x[k+1] = f(x[k],u[k]); solve the weighted projection problem subject to state, input, and terminal constraints; execute u[0]; log p*, solver status, and d = ||p*-z||_W; update the policy using the RL reward. If infeasible, use a preverified backup controller and record the event rather than silently relaxing hard constraints. 3. The control construction computes the feasible trajectory, projection, first action, and terminal-set certificate. Empirically estimate model mismatch, solver failure rates, and the geometry of the feasible set. 4. First experiment: compare PPO with action clipping against PPO plus this layer on cart-pole or a bounded point-mass task with position and acceleration limits. Sweep policy-output scale and measure reward, violation rate, projection distance, and runtime. The prediction is near-zero violations whenever the solver is feasible, d approximately zero for outputs inside the feasible region, and a sharp rise in mean d when output scale crosses the feasibility boundary. For a linear fixed-horizon system, sample the feasible set offline and require online feasibility classification to agree at least 95 percent of the time.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an exact finite-horizon predictive safety layer that projects an unconstrained scalar action onto the state-dependent feasible interval for a bounded point-mass system. The mechanism manifested: feasible outputs had maximum projection distance 0, outside-distance slopes were -1/+1 with error 3.3e-16, the velocity-dependent boundary shift matched theory within 5.6e-17, and all 10,000 feasible random projections were certified. In the stress test clipping had 84 one-step violations while predictive projection had 0; the toy controller also achieved mean reward -51.20 versus -52.77 for clipping, though this is not a broad RL result.

Agent confidence: 9/10

Baseline: Coordinate clipping: mean reward -52.7703, 0.0 violations/episode in the particular 30-episode closed-loop test; 84 violations in 10,000 random one-step stress cases.
Idea: Predictive feasible projection: mean reward -51.2000, 0.0 violations/episode; 0 violations in 10,000 feasible random projections and mean projection distance 3.1837 in the closed-loop test. Core predictions: maximum inside distance 0, outside slope error 3.3e-16, boundary-shift residual 5.6e-17, certified fraction 1.0.

Limitations: This is an exact-model scalar constant-acceleration toy system, not PPO, a neural policy training experiment, or a nonlinear environment. Recursive feasibility across replanning steps, solver failures, model mismatch, differentiability, computational timing, and backup-controller behavior were not tested; the clipping closed-loop scenario happened not to produce violations despite the random stress test exposing its weakness.

How to run: python3 safety_layer_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.23930) Sep 2, 2026

Significantly WORSE than the tuned baseline (+921.0%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0008
Idea mean: 0.0078
Effect (Δ): 0.007 (+921.0%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±75.4%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

The built-in dynamics bench is supervised future-angle prediction rather than closed-loop PPO or actuator execution, so recursive feasibility, solver failures, model mismatch, backup control, and deployment runtime were not tested. The projection uses a conservative scalar bounded-acceleration reachability approximation for the scalar target rather than a full nonlinear pendulum OCP.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented a dynamics-matched feasible-action projection layer around independently trained rnn_small systems, with equal-budget baseline and idea sweeps, 8 paired seeds, and permutation testing. The trained-model mechanism signature was confirmed: projection distance was zero for a subset of predictions and positive outside the reachable interval. However, the safety-layer system had substantially higher test MSE (0.0078135 vs 0.0007652), with paired delta +0.0070482 and p=0.0081, so there was no benchmark win.

Baseline: Test MSE mean 0.0007652444473933429, std 0.00014266442888030616; best config lr=0.006, epochs=18.
Idea: Test MSE mean 0.00781346601434052, std 0.0006524094833102025; best tested config lr=0.001, epochs=18. Paired delta +0.007048221566947177, p=0.0081, 0/8 idea wins.
Mechanism signature: observed_inside_fraction=0.1750, mean_projection_distance=0.0703845, max_projection_distance=0.286004, confirmed=true

Benchmark result