# Entropy-Regularized Wasserstein Actor

- ID: 1946
- Canonical URL: https://synthcore.org/idea/1946/entropy-regularized-wasserstein-actor
- API JSON: https://synthcore.org/api/idea/1946.json
- API Markdown: https://synthcore.org/api/idea/1946.md
- Verification status: beats_baseline
- Source: [arXiv:2608.07433](https://arxiv.org/abs/2608.07433)
- Category: optimization
- Solves: stability, sample-efficiency, accuracy
- ML areas: rl, optimizer, loss
- Math tags: optimal-transport, dynamical-systems, probability, control-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace the usual parameter-space actor update with an action-space transport update. For every visited state, move sampled actions along a critic-improving velocity field while adding the entropy velocity, then fit the transported action cloud back to the actor's Gaussian mean and covariance. This preserves the paper's key idea that policy improvement is a Wasserstein flow over conditional action laws while remaining implementable for neural actors.

## Mathematical statement

The paper's optimal policy class is the state-conditional Gaussian family $\pi_{K,\Sigma}(\cdot\mid x)=\mathcal{N}(-Kx,\Sigma)$, where $x$ is the state, $K$ is the linear feedback gain, and $\Sigma\succ0$ is the action covariance. Its policy-space tangent vectors are represented by action-space continuity equations, $\partial_t\rho_t+\nabla_a\cdot(\rho_t v_t)=0$, where $a$ is the action and $v_t(a,x)$ is the transport velocity. For the entropy-regularized actor objective $\mathcal{J}(\pi)=\mathbb{E}[Q_\phi(x,a)]+\tau\mathbb{E}[\mathcal{H}(\pi(\cdot\mid x))]$, with critic $Q_\phi$, temperature $\tau\ge0$, and differential entropy $\mathcal{H}(\rho)=-\int\rho\log\rho\,da$, the Wasserstein ascent velocity is $v(a,x)=\nabla_aQ_\phi(x,a)-\tau\nabla_a\log\rho(a\mid x)$. For a Gaussian actor $\rho=\mathcal{N}(m_\theta(x),\Sigma_\theta(x))$, $\nabla_a\log\rho(a\mid x)=-\Sigma_\theta(x)^{-1}(a-m_\theta(x))$, giving $v=\nabla_aQ_\phi(x,a)+\tau\Sigma_\theta(x)^{-1}(a-m_\theta(x))$. The first term moves actions toward higher critic value and the second expands low-entropy distributions. The paper proves that this transport is tangent to the Gaussian linear-policy family in LQ systems and produces globally well-posed, exponentially convergent gain/covariance dynamics; the neural adaptation tests whether this local stability benefit survives for nonlinear actors and critics.

## Key formulas

- $$\pi_{K,\Sigma}(\cdot\mid x)=\mathcal{N}(-Kx,\Sigma),\qquad \Sigma\succ0.$$
- $$\partial_t\rho_t(a\mid x)+\nabla_a\!\cdot\big(\rho_t(a\mid x)v_t(a,x)\big)=0.$$
- $$v_t(a,x)=\nabla_a Q_\phi(x,a)-\tau\nabla_a\log\rho_t(a\mid x).$$
- $$\rho_t=\mathcal{N}(m_\theta(x),\Sigma_\theta(x))\quad\Longrightarrow\quad v_t(a,x)=\nabla_aQ_\phi(x,a)+\tau\Sigma_\theta(x)^{-1}\big(a-m_\theta(x)\big).$$

## Implementation notes

Integrate this at the actor update in an off-policy continuous-control algorithm such as SAC, TD3, or a vanilla Gaussian actor-critic. The actor outputs a mean $m_\theta(x)$ and a Cholesky factor $L_\theta(x)$, with covariance $\Sigma_\theta(x)=L_\theta(x)L_\theta(x)^\top+\epsilon I$. For each replay state $x_i$, draw $M$ actions $a_{ij}=m_\theta(x_i)+L_\theta(x_i)z_{ij}$ with $z_{ij}\sim\mathcal{N}(0,I)$. Compute the action-space velocity $v_{ij}=\nabla_aQ_\phi(x_i,a_{ij})+\tau\Sigma_\theta(x_i)^{-1}(a_{ij}-m_\theta(x_i))$, using automatic differentiation for $\nabla_aQ_\phi$. Form transported targets $\tilde a_{ij}=a_{ij}+h v_{ij}$, where $h$ is a small transport step. Fit the actor to these targets by minimizing $\mathcal{L}_{\mathrm{transport}}=\frac{1}{BM}\sum_{i,j}\|m_\theta(x_i)+L_\theta(x_i)z_{ij}-\tilde a_{ij}\|_2^2$, while retaining a covariance-floor penalty if needed. Use one or two gradient steps on this regression loss and stop gradients through the transported targets. Pseudocode: sample replay states; sample Gaussian actions; compute $g=\nabla_aQ$; compute $s=\Sigma^{-1}(a-m)$; set $v=g+\tau s$; set $a_{\mathrm{target}}=a+h v$; update $\theta\leftarrow\theta-\eta_\theta\nabla_\theta\mathcal{L}_{\mathrm{transport}}$. Clip $\|v\|$ and choose $h\|\nabla_aQ\|$ below a fixed fraction of the action range. The paper supplies the continuity-equation/Wasserstein interpretation and Gaussian invariant structure in LQ systems; the neural implementation estimates the critic gradient, discounted state occupancy through replay sampling, and covariance inverses numerically. First compare against SAC on Pendulum-v1 and HalfCheetah-v4 using identical critics, replay buffers, environment interactions, and gradient counts. Measure return versus environment samples, actor-gradient variance, critic-target variance, covariance collapse, and sensitivity to actor learning rate. Success means faster return improvement, fewer entropy-collapse failures, or lower actor-gradient variance at matched final return.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built an entropy-regularized Wasserstein actor toy implementation with critic-gradient transport, entropy velocity, stop-gradient transported targets, and a standard reparameterized actor baseline. The quantitative mechanism checks passed: instability appeared between hλ=1.99 and 2.01, stationary variance matched τ/λ to approximately 1e-15, and observed mean/variance rates matched λ and 2λ within about 0.4%. In the eight-run toy comparison, transport had mean error 1.82e-5 versus 0.00423 and variance error 0.000127 versus 0.00306 for the baseline; this is a promising controlled signal, not evidence of an RL win.

### Mechanism check

- Verdict: Built an entropy-regularized Wasserstein actor toy implementation with critic-gradient transport, entropy velocity, stop-gradient transported targets, and a standard reparameterized actor baseline. The quantitative mechanism checks passed: instability appeared between hλ=1.99 and 2.01, stationary variance matched τ/λ to approximately 1e-15, and observed mean/variance rates matched λ and 2λ within about 0.4%. In the eight-run toy comparison, transport had mean error 1.82e-5 versus 0.00423 and variance error 0.000127 versus 0.00306 for the baseline; this is a promising controlled signal, not evidence of an RL win.
- Confidence: 8/10
- Limitations: Only a one-dimensional quadratic-critic Gaussian toy setting was tested. No Pendulum-v1, HalfCheetah-v4, replay-buffer training, learned critic, multidimensional covariance, actor-gradient variance, wall-clock, or environment sample-efficiency benchmark was run; the baseline and transport losses are also not directly comparable because they optimize different objectives.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/743/bench_report.json)
- [experiment.py](https://synthcore.org/code/743/experiment.py)
- [report.md](https://synthcore.org/code/743/report.md)
- [report_bench_2026-09-01T020636.md](https://synthcore.org/code/743/report_bench_2026-09-01T020636.md)
- [rerun_output.json](https://synthcore.org/code/743/rerun_output.json)
- [results.json](https://synthcore.org/code/743/results.json)
- [stage2_bench.py](https://synthcore.org/code/743/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/743)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.