✗ Failed on benchmark 2026

PPO Correction-Budget Dual Controller

Implementation & benchmark of arXiv:2608.03149 — Process-Knowledge-Embedded Safe DRL for Real-Time Dispatch of Process Loads in Industrial Microgrids

Usefulness8/10

Difficulty5/10

Novelty6/10

Optimization Stability Sample efficiency Accuracy Rl Optimizer Training dynamics Scheduler Optimization Control theory Probability Dynamical systems

Source paper: Process-Knowledge-Embedded Safe DRL for Real-Time Dispatch of Process Loads in Industrial Microgrids arXiv:2608.03149 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Train the raw neural policy to anticipate the safety shield instead of relying on it indefinitely. Add the expected process-correction distance to the PPO objective and adapt its multiplier with a primal-dual update so that policy correction remains below a specified budget.

Formulas

$$C(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^{H-1}\gamma^t c(s_t,a_t)\right],\qquad c(s,a)=\begin{cases}0,&a\in\mathcal{F}(s),\\ d(a,\mathcal{F}(s)),&a\in\mathcal{E}(s),\end{cases}$$

$$\max_\theta\ \mathcal{L}_{\mathrm{PPO}}(\theta)-\lambda\big(C(\theta)-B\big),\qquad \lambda\geq0,$$

$$\lambda_{k+1}=\left[\lambda_k+\rho\big(\widehat C_k-B\big)\right]_+,\qquad \theta_{k+1}=\theta_k+\alpha\nabla_\theta\left[\widehat{\mathcal{L}}_{\mathrm{PPO}}-\lambda_k(\widehat C_k-B)\right],$$

$$\mathbb{E}_{s}\left[\|q(\cdot\mid s)-\pi_\theta(\cdot\mid s)\|_{\mathrm{TV}}\right]\leq\mathbb{E}_{s}[p_E(s)].$$

Mathematical statement

Let c(s,a) be the correction distance incurred when raw action a is excluded or replaced, with c(s,a)=0 for an admissible action. For raw policy \pi_ heta, define discounted correction cost C( heta)=\mathbb{E}_{\pi_ heta}[\sum_{t=0}^{H-1}\gamma^t c(s_t,a_t)] and target budget B. The primal update maximizes the PPO surrogate minus \lambda(C-B), while the dual update is \lambda\leftarrow[\lambda+\rho(C-B)]_+, with multiplier \lambda\geq0 and dual step size \rho>0. For unit correction cost, C is the expected excluded-action probability. The shield dependence is bounded by \|q-\pi\|_{\mathrm{TV}}\leq p_E(s), so reducing C predicts reduced intervention and greater agreement between the raw and executed policies.

Implementation notes

Use the raw actor distribution for PPO ratios and compute a correction cost on every sampled action before applying the active-frontier shield. For each rollout transition, store raw log-probability, shielded action, c(s,a), p_E(s), and feasibility status. Estimate C with the discounted minibatch mean of c; optimize the clipped PPO objective minus lambda times the correction-cost estimate; after each PPO epoch update lambda=max(0,lambda+rho*(C_hat-B)); and optionally cap lambda for numerical stability. The paper supplies the correction-budget and primal-dual mechanism plus the intervention bound; C, advantages, and policy gradients are estimated empirically from rollouts, while feasibility and distance should be computed by an exact simulator. First experiment: PPO on a constrained 10-job, 3-stage scheduling environment, comparing standard PPO with a hard mask against dual-budget PPO with the redistribution shield. Sweep B from zero to the unconstrained correction cost and record cost, violations, C_hat, lambda, and total variation intervention. The prediction is complementary-slackness behavior: if B is achievable, C_hat converges to B or below and lambda stabilizes; if B is below the attainable minimum, lambda grows while C_hat plateaus at that minimum. Mean policy intervention should track logged excluded probability within sampling error.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 7/10

Built a reproducible one-state binary shielded-bandit MVP for the PPO correction-budget dual controller. The primal gradient matched finite differences to 1.9e-13, positive dual updates reduced correction from 0.9971 to 0.0132–0.2260 across rho values, and the infeasible B=0 sweep reduced C from 0.519 to 0.0012 while lambda rose from 1.47 to 3.50. The exact intervention bound held with zero error; however, simultaneous updates showed oscillatory budget tracking, so this is evidence for the mechanism rather than stable PPO-scale convergence.

Agent confidence: 7/10

Baseline: Unconstrained policy-gradient tail correction C=0.9971 at B=0.2; rho=0 produced the same value and lambda=0.
Idea: Dual controller at B=0.2 produced tail C=0.2139, an absolute reduction of 0.7832, with lambda=1.5364. Across budgets, absolute tail budget errors were 0.0139–0.1310 and tail C standard deviations were 0.0846–0.2533 due to discrete-time oscillation. Finite-difference gradient error was 1.85e-13 and TV-bound error was 0.

Limitations: No multi-state scheduling environment, sampled PPO rollouts, neural-network actor, hard-mask baseline, throughput/FLOP comparison, or variance analysis was tested. The toy uses exact expected gradients and simultaneous primal-dual updates, whose stability is sensitive to step sizes.

How to run: python3 toy_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.03149) Sep 2, 2026

Ahead of the baseline (−6.0%) but not statistically significant (p=0.2828, wins 5 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0014
Idea mean: 0.0014
Effect (Δ): -0.0001 (−6.0%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.2828 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±12.1%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

This benchmark is supervised next-state regression rather than on-policy PPO with sampled rollouts, so it tests transfer of the correction-budget primal-dual training mechanism, not the full PPO shielded scheduling system. The shield is an angle-clipping proxy and the correction penalty uses a smooth approximation; no multi-job scheduling environment, exact active-frontier shield, PPO clipping/advantages, or FLOP/speed comparison was tested.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented a matched dynamics-track RNN training modification with a differentiable shield-correction penalty and clipped primal-dual multiplier, using the same architecture, data, epochs, and learning-rate grid as baseline. The idea reduced mean test MSE from 0.00144034 to 0.00135431, but the paired permutation test was non-significant (delta=-0.00008602, p=0.28285). The mechanism prediction was not confirmed: intervention increased from 0.2945 to 0.3000 and smooth correction cost was 0.1045 versus the 0.08 budget.

Baseline: Best lr=0.01; 8-seed test MSE mean=0.0014403378, std=0.0003958336. Sweep means: lr=0.001: 0.0045453925; lr=0.003: 0.0035818806; lr=0.01: 0.0015466219.
Idea: Best lr=0.01, rho=0.15, B=0.08; 8-seed test MSE mean=0.0013543140, std=0.0005574807. Paired delta=-0.0000860238, 5/8 wins, permutation p=0.28285; mean intervention=0.3000000 and smooth correction cost=0.1044550.

Benchmark result