# KKT Actor with Learned Multipliers

- ID: 639
- Canonical URL: https://synthcore.org/idea/639/kkt-actor-with-learned-multipliers
- API JSON: https://synthcore.org/api/idea/639.json
- API Markdown: https://synthcore.org/api/idea/639.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.09461](https://arxiv.org/abs/2607.09461)
- Category: optimization
- Solves: stability, accuracy, sample-efficiency
- ML areas: rl, loss, training, world-model
- Math tags: optimization, convex-analysis, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Train a policy network and separate multiplier networks to minimize Bellman, stationarity, feasibility, and complementarity residuals instead of relying on fixed penalties for constrained actions. The multipliers can turn on only when an inequality constraint binds, allowing one actor to learn both interior and boundary behavior without manually tuning penalty weights.

## Mathematical statement

The paper formulates a Markov control problem with state $s_t\in\mathcal{S}\subseteq\mathbb{R}^{n_s}$, control $c_t\in\mathcal{A}(s_t)\subseteq\mathbb{R}^{n_c}$, Markov policy $c_t=c(s_t)$, and transition $s_{t+1}=\psi(s_t,c_t,z_{t+1})$, where $z_{t+1}$ is an exogenous innovation. For a differentiable action-value objective $Q_\theta(s,c)$, equality constraints $h_\phi(s,c)=0$ and inequality constraints $g_\phi(s,c)\le 0$, use the KKT Lagrangian $\mathcal{L}(s,c,\lambda,\mu)=-Q_\theta(s,c)+\lambda^\top g_\phi(s,c)+\mu^\top h_\phi(s,c)$. The KKT conditions are stationarity $r_{\mathrm{stat}}=\nabla_c\mathcal{L}=0$, primal feasibility $r_h=h_\phi(s,c)=0$ and $r_g=\max(g_\phi(s,c),0)=0$, dual feasibility $\lambda\ge0$, and complementarity $\lambda_i g_i=0$. Here $\lambda$ are nonnegative inequality multipliers and $\mu$ are unrestricted equality multipliers. Parameterize $c=\pi_\omega(s)$, $\lambda=\operatorname{softplus}(\rho_\eta(s))$, and $\mu=\nu_\xi(s)$. A numerically smoother complementarity residual is the Fischer-Burmeister function $\operatorname{FB}(a,b)=\sqrt{a^2+b^2+\varepsilon^2}-a-b$, applied as $r_{\mathrm{comp},i}=\operatorname{FB}(\lambda_i,-g_i)$. The training objective is $\mathbb{E}_s[\|r_{\mathrm{stat}}\|_2^2+\alpha_h\|r_h\|_2^2+\alpha_g\|r_g\|_2^2+\alpha_c\|r_{\mathrm{comp}}\|_2^2+\alpha_B r_B^2]$, where $r_B$ is the Bellman or critic residual.

## Key formulas

- $$s_t\in\mathcal{S}\subseteq\mathbb{R}^{n_s},\qquad c_t\in\mathcal{A}(s_t)\subseteq\mathbb{R}^{n_c},\qquad c_t=c(s_t),$$
- $$s_{t+1}=\psi(s_t,c_t,z_{t+1}),$$
- $$\mathcal{L}(s,c,\lambda,\mu)=-Q_\theta(s,c)+\lambda^\top g_\phi(s,c)+\mu^\top h_\phi(s,c),$$
- $$r_{\mathrm{stat}}=\nabla_c\mathcal{L},\quad r_h=h_\phi(s,c),\quad r_g=\max(g_\phi(s,c),0),\quad r_{\mathrm{comp},i}=\sqrt{\lambda_i^2+(-g_i)^2+\varepsilon^2}-\lambda_i+g_i.$$

## Implementation notes

(1) Integration point: use a continuous-action actor-critic or model-based RL system. Keep the actor $\pi_\omega(s)$ unchanged at the interface, but add an inequality-multiplier head $\rho_\eta(s)$ and an equality-multiplier head $\nu_\xi(s)$. The critic $Q_\theta(s,c)$ must be differentiable with respect to its action input. Constraints may be environment limits such as $c\ge0$, a budget $\sum_i c_i\le B$, torque limits, or safety functions $g(s,c)\le0$.

(2) Pseudocode:
```text
for minibatch of states s:
    c = actor(s)
    lam = softplus(multiplier_ineq(s))
    mu = multiplier_eq(s)
    q = critic(s, c)
    L = -q + dot(lam, g(s,c)) + dot(mu, h(s,c))
    r_stat = grad(L, c)
    r_h = h(s,c)
    r_g = relu(g(s,c))
    r_comp = sqrt(lam**2 + (-g(s,c))**2 + eps**2) - lam + g(s,c)
    r_B = reward(s,c) + gamma * V_target(next_state) - V(s)
    loss = norm(r_stat)**2 + ah*norm(r_h)**2 + ag*norm(r_g)**2
           + ac*norm(r_comp)**2 + aB*r_B**2
    update actor, multiplier heads, critic using loss
```
The stationarity expression is differentiated by automatic differentiation; use `create_graph=True` so actor gradients include derivatives of $Q$ and the constraints. (3) The KKT identities are fixed by the mathematics. Estimate $Q$, $V$, constraint Jacobians, and transition targets from the critic and minibatch data. Normalize each residual by a running RMS and ramp $\alpha_c$ from zero to its target value over 5,000 updates. Optionally clip multiplier outputs to a maximum such as 100 to prevent unstable early gradients. (4) First cheap experiment: compare SAC with a standard fixed penalty, SAC with a scalar Lagrange multiplier update, and this KKT actor on a constrained Pendulum or Safety-Gym task with one equality and one occasionally active inequality. Use identical replay buffers and network sizes. Measure constraint violation, return, boundary-state performance, and sensitivity to penalty coefficients at equal environment interactions. Success is lower 95th-percentile violation with equal or higher return, especially near the constraint boundary, and less sensitivity to penalty hyperparameters.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built and ran a scalar constrained actor with a learned softplus multiplier head, Fischer-Burmeister complementarity residual, stationarity residual, and fixed-penalty baselines. The mechanism manifested quantitatively: the active multiplier had the predicted slope 2.0 versus 2.0, inactive multipliers were analytically zero, exact KKT actions had zero violation, and fixed-penalty violation ratios matched 1/(1+P) exactly for P=0.1, 1, and 10. In the learned mini-experiment, KKT training achieved max violation 0.0057 versus 0.0900 even for penalty 10 and 0.9052 for penalty 0.1, with much lower action MSE than all tested baselines.

### Mechanism check

- Verdict: Built and ran a scalar constrained actor with a learned softplus multiplier head, Fischer-Burmeister complementarity residual, stationarity residual, and fixed-penalty baselines. The mechanism manifested quantitatively: the active multiplier had the predicted slope 2.0 versus 2.0, inactive multipliers were analytically zero, exact KKT actions had zero violation, and fixed-penalty violation ratios matched 1/(1+P) exactly for P=0.1, 1, and 10. In the learned mini-experiment, KKT training achieved max violation 0.0057 versus 0.0900 even for penalty 10 and 0.9052 for penalty 0.1, with much lower action MSE than all tested baselines.
- Confidence: 9/10
- Limitations: This is a deterministic scalar toy problem, not a full SAC, Pendulum, or Safety-Gym environment. It does not test replay-buffer sample efficiency, Bellman or critic residuals, equality constraints, stochastic transitions, 95th-percentile violations across trajectories, or sensitivity across random seeds. The softplus multiplier cannot attain exact zero and the Fischer-Burmeister epsilon introduces a small residual floor.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/287/bench_report.json)
- [kkt_bench.py](https://synthcore.org/code/287/kkt_bench.py)
- [kkt_experiment.py](https://synthcore.org/code/287/kkt_experiment.py)
- [report.md](https://synthcore.org/code/287/report.md)
- [report_bench_2026-08-31T001633.md](https://synthcore.org/code/287/report_bench_2026-08-31T001633.md)
- [results.json](https://synthcore.org/code/287/results.json)
- [Download all files as ZIP](https://synthcore.org/download/287)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.