# PPO Correction-Budget Dual Controller

- ID: 1741
- Canonical URL: https://synthcore.org/idea/1741/ppo-correction-budget-dual-controller
- API JSON: https://synthcore.org/api/idea/1741.json
- API Markdown: https://synthcore.org/api/idea/1741.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.03149](https://arxiv.org/abs/2608.03149)
- Category: optimization
- Solves: stability, sample-efficiency, accuracy
- ML areas: rl, optimizer, training-dynamics, scheduler
- Math tags: optimization, control-theory, probability, dynamical-systems
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Train the raw neural policy to anticipate the safety shield instead of relying on it indefinitely. Add the expected process-correction distance to the PPO objective and adapt its multiplier with a primal-dual update so that policy correction remains below a specified budget.

## Mathematical statement

Let c(s,a) be the correction distance incurred when raw action a is excluded or replaced, with c(s,a)=0 for an admissible action. For raw policy \pi_    heta, define discounted correction cost C(    heta)=\mathbb{E}_{\pi_    heta}[\sum_{t=0}^{H-1}\gamma^t c(s_t,a_t)] and target budget B. The primal update maximizes the PPO surrogate minus \lambda(C-B), while the dual update is \lambda\leftarrow[\lambda+\rho(C-B)]_+, with multiplier \lambda\geq0 and dual step size \rho>0. For unit correction cost, C is the expected excluded-action probability. The shield dependence is bounded by \|q-\pi\|_{\mathrm{TV}}\leq p_E(s), so reducing C predicts reduced intervention and greater agreement between the raw and executed policies.

## Key formulas

- $$C(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^{H-1}\gamma^t c(s_t,a_t)\right],\qquad c(s,a)=\begin{cases}0,&a\in\mathcal{F}(s),\\ d(a,\mathcal{F}(s)),&a\in\mathcal{E}(s),\end{cases}$$
- $$\max_\theta\ \mathcal{L}_{\mathrm{PPO}}(\theta)-\lambda\big(C(\theta)-B\big),\qquad \lambda\geq0,$$
- $$\lambda_{k+1}=\left[\lambda_k+\rho\big(\widehat C_k-B\big)\right]_+,\qquad \theta_{k+1}=\theta_k+\alpha\nabla_\theta\left[\widehat{\mathcal{L}}_{\mathrm{PPO}}-\lambda_k(\widehat C_k-B)\right],$$
- $$\mathbb{E}_{s}\left[\|q(\cdot\mid s)-\pi_\theta(\cdot\mid s)\|_{\mathrm{TV}}\right]\leq\mathbb{E}_{s}[p_E(s)].$$

## Implementation notes

Use the raw actor distribution for PPO ratios and compute a correction cost on every sampled action before applying the active-frontier shield. For each rollout transition, store raw log-probability, shielded action, c(s,a), p_E(s), and feasibility status. Estimate C with the discounted minibatch mean of c; optimize the clipped PPO objective minus lambda times the correction-cost estimate; after each PPO epoch update lambda=max(0,lambda+rho*(C_hat-B)); and optionally cap lambda for numerical stability. The paper supplies the correction-budget and primal-dual mechanism plus the intervention bound; C, advantages, and policy gradients are estimated empirically from rollouts, while feasibility and distance should be computed by an exact simulator. First experiment: PPO on a constrained 10-job, 3-stage scheduling environment, comparing standard PPO with a hard mask against dual-budget PPO with the redistribution shield. Sweep B from zero to the unconstrained correction cost and record cost, violations, C_hat, lambda, and total variation intervention. The prediction is complementary-slackness behavior: if B is achievable, C_hat converges to B or below and lambda stabilizes; if B is below the attainable minimum, lambda grows while C_hat plateaus at that minimum. Mean policy intervention should track logged excluded probability within sampling error.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built a reproducible one-state binary shielded-bandit MVP for the PPO correction-budget dual controller. The primal gradient matched finite differences to 1.9e-13, positive dual updates reduced correction from 0.9971 to 0.0132–0.2260 across rho values, and the infeasible B=0 sweep reduced C from 0.519 to 0.0012 while lambda rose from 1.47 to 3.50. The exact intervention bound held with zero error; however, simultaneous updates showed oscillatory budget tracking, so this is evidence for the mechanism rather than stable PPO-scale convergence.

### Mechanism check

- Verdict: Built a reproducible one-state binary shielded-bandit MVP for the PPO correction-budget dual controller. The primal gradient matched finite differences to 1.9e-13, positive dual updates reduced correction from 0.9971 to 0.0132–0.2260 across rho values, and the infeasible B=0 sweep reduced C from 0.519 to 0.0012 while lambda rose from 1.47 to 3.50. The exact intervention bound held with zero error; however, simultaneous updates showed oscillatory budget tracking, so this is evidence for the mechanism rather than stable PPO-scale convergence.
- Confidence: 7/10
- Limitations: No multi-state scheduling environment, sampled PPO rollouts, neural-network actor, hard-mask baseline, throughput/FLOP comparison, or variance analysis was tested. The toy uses exact expected gradients and simultaneous primal-dual updates, whose stability is sensitive to step sizes.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.28285
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/552/bench_report.json)
- [report.md](https://synthcore.org/code/552/report.md)
- [report_bench_2026-08-31T174014.md](https://synthcore.org/code/552/report_bench_2026-08-31T174014.md)
- [results.json](https://synthcore.org/code/552/results.json)
- [stage2_bench.py](https://synthcore.org/code/552/stage2_bench.py)
- [toy_experiment.py](https://synthcore.org/code/552/toy_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/552)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.