△ Mechanism confirmed, baseline not beaten 2026

Feasibility-Ranked Group Policy Gradient

Implementation & benchmark of arXiv:2608.15359 — Ranking-Augmented On-Policy Optimization with Adaptive Advantage-Normalization for Constrained Control

Usefulness7/10

Difficulty4/10

Novelty6/10

Dynamics Stability Sample efficiency Rl Optimizer Transformer Training dynamics Optimization Probability Statistics Control theory Dynamical systems

Source paper: Ranking-Augmented On-Policy Optimization with Adaptive Advantage-Normalization for Constrained Control arXiv:2608.15359 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a learned critic with group-relative trajectory advantages whose weights are explicitly ordered by terminal feasibility. Feasible rollouts receive larger positive update weight than violating rollouts, while per-timestep normalization prevents high-variance late-horizon returns from dominating the policy gradient.

Formulas

$$\widetilde A_{b,t}=\frac{A_{b,t}-\mu_t}{\sigma_t+\varepsilon},\qquad \mu_t=\frac{1}{B}\sum_{b=1}^{B}A_{b,t},\qquad \sigma_t^2=\frac{1}{B}\sum_{b=1}^{B}(A_{b,t}-\mu_t)^2.$$

$$L_{\mathrm{rank\text{-}clip}}(\theta)=\frac{1}{BT}\sum_{b=1}^{B}\sum_{t=0}^{T-1}w_b\min\left(\rho_{b,t}\widetilde A_{b,t},\operatorname{clip}(\rho_{b,t},1-\epsilon,1+\epsilon)\widetilde A_{b,t}\right).$$

$$w_F\,\mathbb E[\widetilde A_t\mid F=1]>w_V\,\mathbb E[\widetilde A_t\mid F=0]\quad\Longrightarrow\quad \text{the expected update has a positive feasible-trajectory component}.$$

Mathematical statement

For a batch of B trajectories of horizon T, let G_b be the return of trajectory b, let v_b >= 0 be its terminal constraint violation, and let F_b = 1[v_b <= delta] denote feasibility. Define a raw group-relative advantage A_{b,t} = G_b - B^{-1} sum_j G_j or a return-to-go analogue. Normalize independently at each timestep using A_tilde_{b,t} = (A_{b,t} - mu_t)/(sigma_t + epsilon), where mu_t = B^{-1} sum_b A_{b,t} and sigma_t^2 = B^{-1} sum_b (A_{b,t} - mu_t)^2. Introduce ranking weights w_b = w_F for feasible trajectories and w_b = w_V for violating trajectories, with w_F > w_V >= 0, or use a monotone rank function of v_b. The policy update maximizes the clipped objective with importance ratio rho_{b,t} = pi_theta(a_{b,t}|o_{b,t}) / pi_theta_old(a_{b,t}|o_{b,t}). The normalization controls per-timestep group variance, while the ranking mechanism has a positive expected feasibility bias whenever w_F E[A_tilde_t | F = 1] > w_V E[A_tilde_t | F = 0].

Implementation notes

Integrate this at the PPO/GRPO minibatch construction stage for an autoregressive Transformer policy or recurrent control policy. Collect B complete rollouts with the current policy, retaining observations, actions, rewards, old log probabilities, and the terminal constraint quantity. Compute each trajectory return G_b and terminal violation v_b = max(0, |c(x_{b,T}) - c_star| - delta); set F_b = 1 when v_b = 0. For every timestep, form group-relative raw advantages, compute the batch mean and standard deviation across trajectories, and normalize with a small epsilon. Choose w_b = w_F for feasible trajectories and w_b = w_V otherwise, initially using (w_F,w_V) = (2,1); optionally interpolate weights continuously with violation rank. Then optimize the clipped objective for several epochs, recomputing neither the normalization nor old-policy ratios during those epochs. Pseudocode is: rollout; compute G,v,F; compute A,mu_t,sigma_t,A_tilde; assign w; evaluate rho; backpropagate -L_rank-clip. The paper supplies the normalization and ranking mechanism; empirical estimates are mu_t, sigma_t, and conditional gradient statistics. First test on constrained CartPole or Pendulum with a terminal-state target, comparing PPO, critic-free GRPO, and this method at equal environment steps. Log normalized variance, feasible and violating weighted advantage means, and sustained feasibility. The quantitative prediction is that each timestep's normalized group variance remains approximately one and that the measured feasible-update margin w_F Abar_F - w_V Abar_V becomes positive before sustained feasibility improves. If the margin is negative, increasing ranking strength should not help and indicates that the ranking condition is unmet.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and ran a self-contained feasibility-ranked group policy-gradient MVP with timestep-wise normalization, clipped ratios, and feasible/violating trajectory weights. The mechanism manifested: normalized variance matched the exact prediction across scales (0.9999996–0.999999999), and observed feasibility-margin values tracked analytical predictions closely across return separations and weight ratios. In the toy policy experiment, ranking increased final feasibility from 2.20e-9 to 3.21e-5 and reduced risky-action probability from 0.9815 to 0.8741, at the expected cost of lower unconstrained reward.

Agent confidence: 8/10

Baseline: Standard normalized group-relative update: final feasibility 2.20e-9, risky probability 0.9815, expected reward 5.9629.
Idea: Feasibility-ranked update (wF=3, wV=0.25): final feasibility 3.21e-5, risky probability 0.8741, expected reward 5.7481. Normalized variance prediction error was below 4e-7; margin predictions were generally within about 2% for nonzero separation.

Limitations: The policy comparison is a tiny synthetic Bernoulli horizon rather than CartPole/Pendulum, and does not compare against a learned-critic PPO implementation. The feasibility gain is absolute but both methods remain poor under the deliberately conflicting reward/constraint setup; no larger-scale sample-efficiency or wall-clock study was performed.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.15359) Sep 2, 2026

Ahead of the baseline (−62.2%) but not statistically significant (p=0.0533, wins 6 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0039
Idea mean: 0.0015
Effect (Δ): -0.0024 (−62.2%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.0533 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±56.6%
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 9 configs

Limitations:

Only the built-in dynamics regression track was tested; no CartPole/Pendulum environment, learned-critic PPO comparison, longer training budget, larger seed count, or independent replication was run. The custom surrogate uses stochastic prediction groups and a small supervised anchor because the benchmark exposes supervised sequence regression rather than rollout interaction.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 9/10

Built a local feasibility-ranked group policy-gradient surrogate on the structurally matched dynamics/rnn_small benchmark, with identical architecture and standard test-MSE evaluation for both systems. Mean MSE improved from 0.00393015 to 0.00148475 with 6/8 paired wins, and the model-derived mechanism signature was confirmed. The paired permutation p-value was 0.0533, so the required significant-win criterion was not met.

Baseline: Best baseline lr=0.003, mean test MSE=0.0039301491, std=0.0019852212.
Idea: Best ranked setting wf=2.0,wv=1.0,lr=0.003, mean test MSE=0.0014847495, std=0.0005846185; delta_mean=-0.0024453995; 6/8 wins; permutation p=0.0533.
Mechanism signature: Observed normalized advantage variance=0.9999971 and weighted feasible margin=1.6912; confirmed=true.

Benchmark result