△ Mechanism confirmed, baseline not beaten 2026

Coordinate Path-Integral Joint Gibbs Policy

Implementation & benchmark of arXiv:2607.19928 — Continuous-Time Reinforcement Learning for $N$-Player Stochastic Differential Games with Exploratory Policies

Usefulness7/10

Difficulty6/10

Novelty8/10

Sampling Stability Accuracy Sample efficiency Rl Sampling Moe routing Game theory Calculus Probability Geometry

Source paper: Continuous-Time Reinforcement Learning for $N$-Player Stochastic Differential Games with Exploratory Policies arXiv:2607.19928 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Construct a joint exploratory policy directly from the players' learned q-functions even when their Gibbs conditionals are incompatible. Integrate the players' own-action gradients along a fixed coordinate path to obtain a scalar joint energy, then sample all actions from one tempered Gibbs distribution; this supplies a coherent correlated exploration mechanism rather than independently sampling contradictory policies.

Formulas

$$\Phi_{1\to N}(x,u)=\sum_{k=1}^{N}\int_{u_k^0}^{u_k}F_k\!\left(x,u_1,\ldots,u_{k-1},s,u_{k+1}^0,\ldots,u_N^0\right)\,ds,\qquad F_k=\partial_{u_k}q_k.$$

$$p_{\mathrm{path}}(u\mid x)=\frac{\exp\left(\Phi_{1\to N}(x,u)/\gamma\right)}{Z_{\mathrm{path}}(x)},\qquad Z_{\mathrm{path}}(x)=\int_{U^N}\exp\left(\Phi_{1\to N}(x,v)/\gamma\right)\,dv.$$

$$\Delta_{\pi,\pi'}(x,u):=\Phi_{\pi}(x,u)-\Phi_{\pi'}(x,u),\qquad \Delta_{\pi,\pi'}\equiv0\ \text{if}\ \partial_{u_j}F_i=\partial_{u_i}F_j\ \forall i\neq j.$$

Mathematical statement

Let q_i(x,u) be the player-i action value, γ>0 the exploration temperature, and F_i(x,u)=∂_{u_i}q_i(x,u). Choose a reference action u^0 and a fixed coordinate path that changes u_1,...,u_N sequentially. The path-integral energy is Φ_path(x,u)=Σ_{k=1}^N∫_{u_k^0}^{u_k}F_k(x,u_1,...,u_{k-1},s,u_{k+1}^0,...,u_N^0)ds. If the cross-partial condition holds, this integral is path independent and its gradient equals F. If it fails, the fixed path still defines an explicit approximate correlated equilibrium. The resulting joint policy is p_path(u|x)=exp(Φ_path(x,u)/γ)/Z_path(x), with Z_path the integral over the bounded action domain. Disagreement between different coordinate orders is an incompatibility diagnostic: ΔΦ=Φ_path-Φ_path' is zero for a conservative field.

Implementation notes

Use this as the exploration/action-sampling layer in a centralized-training multi-agent actor, while retaining decentralized actors for execution if required. Restrict each action to a bounded box, choose u0 as the midpoint, and evaluate the learned critics q_i and their own-action gradients F_i with automatic differentiation. For each sampled candidate action u, approximate each one-dimensional integral in Phi with K-point Gauss-Legendre quadrature or a trapezoidal sum along the corresponding coordinate segment. Compute logits Phi(u)/gamma for M joint candidates generated by low-discrepancy points or a factorized proposal, normalize them with log-sum-exp, and sample a candidate; alternatively use Metropolis-Hastings with target density exp(Phi/gamma) if N or action dimension is larger. During training, include the selected joint action in the replay buffer and use the same energy policy for target-action generation. Randomly permute coordinate orders during diagnostics and optionally average their energies to reduce path bias. The path integral and Gibbs normalization are computed from the paper's construction; quadrature order, proposal distribution, and temperature schedule are empirical. First test on two-agent continuous particle or MPE cooperative navigation against independent Gaussian exploration and a centralized Gaussian actor. Measure collision rate, return, action mutual information, KL between conditional marginals and each agent's Gibbs response, and wall-clock sampling cost. Success is improved coordination and return at the same number of environment steps, with path-order disagreement decreasing during training.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a readable two-agent coordinate-path Gibbs MVP with bounded-box quadrature, joint sampling, independent Gibbs baseline, path-order diagnostics, and finite-difference verification. The core predictions were confirmed: disagreement RMS matched |c-b|/3 within 0.5%, conservative fields had exactly zero path difference, and the gradient error was 2.9e-12. The coherent sampler also produced positive coordination (mean u1*u2=0.0618 versus -0.00048 independently), but this is only a toy signal, not an RL performance win.

Agent confidence: 8/10

Baseline: Independent bounded Gibbs: mean product -0.00048; mean absolute product 0.13112.
Idea: Coordinate-path joint Gibbs: mean product 0.06176; mean absolute product 0.14438. Path-disagreement RMS observed/predicted: 0, 0.0670/0.0667, 0.1675/0.1667, 0.3350/0.3333, 0.5025/0.5000. Gradient max error 2.86e-12.

Limitations: Only a two-dimensional analytic toy field was tested; no neural critics, automatic differentiation, multi-agent environment, replay training, RL return, collision rate, KL metric, runtime scaling, or GPU implementation was evaluated. The unbounded quadratic Gibbs covariance prediction was intentionally compared against a bounded domain and diverged at high temperature because boundary effects dominate.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.19928) Sep 2, 2026

Significantly WORSE than the tuned baseline (+294.2%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Constrained_two_agent_dynamics
Model: custom_gru_cell
Paired seeds: 8
Baseline mean: 0.0063
Idea mean: 0.0247
Effect (Δ): 0.0184 (+294.2%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±111.1%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

The registered track is a constrained two-agent dynamics forecasting task rather than a full multi-agent RL replay environment, so exploration return, collision rate, and runtime scaling were not tested. The path-integral intervention was implemented as a differentiable consistency loss over recurrent-model gradients; the measured signature confirms near-zero path disagreement but does not establish a performance benefit.

How to run: python3 stage2_registered.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

The Coordinate Path-Integral Joint Gibbs intervention was evaluated on the registered constrained_two_agent_dynamics track. It significantly worsened standard test MSE: baseline mean 0.0062669871 versus idea mean 0.0247058901, paired delta +0.0184389030, permutation p=0.0081. The trained-model path-order signature measured near-zero disagreement, but the mechanism did not improve forecasting performance.

Baseline: Best configuration lr=0.006, epochs=12, hidden=32, wd=0.0; full mean MSE 0.006266987067647278, std 0.0012038076406143412, per-seed [0.0054755136370658875, 0.008236748166382313, 0.007714319974184036, 0.006880192551761866, 0.005664798431098461, 0.006498826202005148, 0.004625890403985977, 0.005039607174694538].
Idea: Best configuration lr=0.006, epochs=12, hidden=32, wd=0.0, lam=0.2, K=7; mean MSE 0.024705890100449324, std 0.008283505340451159, per-seed [0.01789645291864872, 0.03382120281457901, 0.017041007056832314, 0.02942308597266674, 0.03449169918894768, 0.03302520141005516, 0.012732679024338722, 0.019215792417526245].

Benchmark result