One-Step Saddle Deviation Regularizer

Integrate this at the policy-loss and critic-loss interface in a two-player zero-sum RL system, self-play environment, adversarial data-generation model, or generator-discriminator variant. Let the two policy networks output logits for state s, producing distributions \pi_P(s) and \pi_B(s); let a centralized critic or one-step simulator produce a payoff matrix M(s)\in\mathbb R^{|A_P| imes |A_B|}. First compute an approximate saddle point by 5-20 projected-gradient iterations: p\leftarrow\operatorname{softmax}(\log p+\eta Mq), q\leftarrow\operatorname{softmax}(\log q-\eta M^{\mathsf T}p), with p initialized from the Pitcher policy and q from the Batter policy. Estimate V(s)=p^{\mathsf T}M(s)q. Then compute all row values Q_P[a]=(M(s)q)_a and all column values Q_B[b]=(p^{\mathsf T}M(s))_b. Form G_P[a]=Q_P[a]-V and G_B[b]=V-Q_B[b], clamp at zero, and add \lambda\mathcal L_{\mathrm{dev}} to the corresponding policy objective. Detach M and the approximate saddle point during the first experiment to avoid unstable higher-order gradients; later compare with differentiating through the saddle iterations. M(s) can be obtained from exact one-step rollouts, a learned critic, or bootstrap targets r+\gamma V(s'). The math supplies the forced-action evaluations and nonnegativity target; the critic and saddle solver are estimated empirically. Start with a small two-agent matrix-game benchmark and a 5x5 gridworld with delayed rewards, comparing PPO/self-play without the penalty against the same method with \lambda\in\{0.01,0.1,1\}. Measure exploitability, policy oscillation, return, and critic error. Success means lower exploitability and smoother value curves at equal environment interactions, without reducing final average return.