# Cross-Partial Nash Compatibility Regularizer

- ID: 1167
- Canonical URL: https://synthcore.org/idea/1167/cross-partial-nash-compatibility-regularizer
- API JSON: https://synthcore.org/api/idea/1167.json
- API Markdown: https://synthcore.org/api/idea/1167.md
- Verification status: beats_baseline
- Source: [arXiv:2607.19928](https://arxiv.org/abs/2607.19928)
- Category: regularization
- Solves: stability, accuracy, sample-efficiency
- ML areas: rl, loss, regularization
- Math tags: game-theory, calculus, geometry
- Ratings: usefulness 7/10; difficulty 5/10; novelty 8/10

## Idea description

Add an integrability penalty to a multi-agent critic so that the players' entropy-regularized Gibbs best responses can be represented by one coherent joint policy. The penalty detects whether the learned action-value functions define a conservative joint action field, preventing independent agents from learning mutually incompatible conditional policies.

## Mathematical statement

For players i=1,...,N with continuous scalar actions u_i and learned q-functions q_i(x,u_1,...,u_N), the entropy-regularized conditional best response has the Gibbs form π_i(u_i|u_{-i},x) proportional to exp(q_i(x,u)/γ), where γ>0 is the exploration temperature and u_{-i} denotes all actions except u_i. Define the action field F_i(x,u)=∂q_i(x,u)/∂u_i. A necessary compatibility condition for these conditionals to arise from one differentiable joint energy Φ is the conservative-field identity ∂F_i/∂u_j=∂F_j/∂u_i for every i≠j, equivalently ∂²q_i/(∂u_j∂u_i)=∂²q_j/(∂u_i∂u_j). Penalize violations with L_compat=E[Σ_{i<j}(∂²q_i/(∂u_j∂u_i)-∂²q_j/(∂u_i∂u_j))²]. On a simply connected action domain, a zero residual makes the field conservative, so a scalar joint Gibbs energy can be constructed.

## Key formulas

- $$\pi_i(u_i\mid u_{-i},x)=\frac{\exp\left(q_i(x,u)/\gamma\right)}{\int_U\exp\left(q_i(x,(v_i,u_{-i}))/\gamma\right)\,dv_i},\qquad \gamma>0.$$
- $$F_i(x,u):=\frac{\partial q_i(x,u)}{\partial u_i},\qquad \frac{\partial F_i}{\partial u_j}=\frac{\partial F_j}{\partial u_i}\quad(i\neq j).$$
- $$\mathcal L_{\mathrm{compat}}=\mathbb E_{(x,u)\sim\mathcal B}\left[\sum_{i<j}\left(\frac{\partial^2 q_i}{\partial u_j\partial u_i}-\frac{\partial^2 q_j}{\partial u_i\partial u_j}\right)^2\right],\qquad \mathcal L=\mathcal L_{\mathrm{RL}}+\lambda_{\mathrm{compat}}\mathcal L_{\mathrm{compat}}.$$

## Implementation notes

Integrate this into the centralized critic of a continuous-control multi-agent algorithm such as MADDPG, MASAC, or MAPPO. The critic should expose q_i(x,u_1,...,u_N) for every player, with action inputs kept differentiable. For each minibatch of states and joint actions, compute first derivatives g_i=autograd.grad(q_i,u_i,create_graph=True); compute h_ij=autograd.grad(g_i,u_j,create_graph=True); accumulate the squared residual (h_ij-h_ji)^2 over i<j; and add lambda_compat times this quantity to the critic loss. Use Hutchinson probes for high-dimensional vector actions: estimate the mixed-Jacobian discrepancy with random Rademacher vectors rather than materializing every Hessian entry. Pseudocode is: sample batch B; evaluate q_i; for each i compute g_i=dq_i/du_i; for each pair compute h_ij=d g_i/du_j and h_ji=d g_j/du_i; Lc=sum(norm(h_ij-h_ji)^2); update critic on L_RL+lambda Lc; update actors normally using their critics. The mathematical quantity is the cross-partial residual; lambda, action smoothing, and whether to apply the penalty only near replay actions are empirical choices. First test on a two-agent, two-dimensional-action particle-navigation or HalfCheetah coordination task against MADDPG/MASAC. Track critic Bellman error, variance of actor gradients, achieved return, and the empirical residual. Success means lower gradient variance and higher return at equal environment interactions, without a substantial increase in critic wall-clock cost.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: helps
- Verdict: Built and numerically verified an autograd cross-partial compatibility regularizer for a two-agent centralized-critic toy problem. The autograd residual matched an independent finite-difference calculation to 2.0e-12; the residual followed the predicted 2/(1+4λ) law across the lambda sweep, and the Gibbs incompatibility squared followed the predicted 1/γ² law exactly. Regularization reduced the residual from 2.0 to 0.4 at λ=1, while critic fitting loss increased from approximately 0 to 0.64 because the deliberately incompatible target critics cannot both be preserved under a conservative-field constraint; this confirms the mechanism, but not an RL return improvement.

### Mechanism check

- Verdict: Built and numerically verified an autograd cross-partial compatibility regularizer for a two-agent centralized-critic toy problem. The autograd residual matched an independent finite-difference calculation to 2.0e-12; the residual followed the predicted 2/(1+4λ) law across the lambda sweep, and the Gibbs incompatibility squared followed the predicted 1/γ² law exactly. Regularization reduced the residual from 2.0 to 0.4 at λ=1, while critic fitting loss increased from approximately 0 to 0.64 because the deliberately incompatible target critics cannot both be preserved under a conservative-field constraint; this confirms the mechanism, but not an RL return improvement.
- Confidence: 9/10
- Limitations: This is a controlled two-agent scalar-action critic-fitting MVP, not a full MADDPG, MASAC, or environment experiment. It does not measure environment return, actor-gradient variance, replay-buffer sample efficiency, Hutchinson estimation, or wall-clock overhead in a high-dimensional neural critic.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/465/bench_report.json)
- [bench_run.py](https://synthcore.org/code/465/bench_run.py)
- [compat_track.py](https://synthcore.org/code/465/compat_track.py)
- [experiment.py](https://synthcore.org/code/465/experiment.py)
- [report.md](https://synthcore.org/code/465/report.md)
- [report_bench_2026-08-31T125413.md](https://synthcore.org/code/465/report_bench_2026-08-31T125413.md)
- [results.json](https://synthcore.org/code/465/results.json)
- [Download all files as ZIP](https://synthcore.org/download/465)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.