# Entropic Wasserstein adversarial augmentation

- ID: 747
- Canonical URL: https://synthcore.org/idea/747/entropic-wasserstein-adversarial-augmentation
- API JSON: https://synthcore.org/api/idea/747.json
- API Markdown: https://synthcore.org/api/idea/747.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.11460](https://arxiv.org/abs/2607.11460)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: loss, data-augmentation, training-dynamics
- Math tags: optimal-transport, probability, convex-analysis
- Ratings: usefulness 8/10; difficulty 4/10; novelty 6/10

## Idea description

Replace the nonsmooth Wasserstein inner supremum in robust training by the paper's entropic log-expectation, evaluated with Gaussian perturbation samples. The resulting loss continuously interpolates between ordinary averaging and soft worst-case selection, producing differentiable adversarial augmentation without an inner PGD loop.

## Mathematical statement

For a decision variable x, scenario xi, transport cost c(xi,zeta), dual radius multiplier lambda >= 0, and smoothing temperature epsilon > 0, the paper defines the sample-wise adversarial value phi_epsilon(x,lambda,xi) = epsilon log E[exp((f(x,zeta)-lambda c(xi,zeta))/epsilon)], where zeta is sampled from N(xi,sigma^2 I). The smoothed WDRO objective is inf_{lambda >= 0} lambda rho + E_{xi_hat ~ P_hat_N}[phi_epsilon(x,lambda,xi_hat)], where rho is the Wasserstein ambiguity radius and P_hat_N is the empirical training distribution. Proposition 3 defines m_c = max_k E[c(xi_hat_k,zeta)] and, when rho > m_c, restricts lambda to [0, lambda_max], with lambda_max = 2 ||f||_infty / (rho - m_c). For neural training, set x = theta, f(theta,zeta) = ell(g_theta(zeta),y), and approximate the expectation with K Monte Carlo samples. The gradient is a soft adversarial average, weighted by exp((loss - lambda times transport cost)/epsilon).

## Key formulas

- $$\varphi_{\varepsilon}(\mathbf{x},\lambda,\xi)=\varepsilon\log\mathbb{E}_{\zeta\sim\mathcal{N}(\xi,\sigma^{2}\mathbf{I})}\left[\exp\left(\frac{f(\mathbf{x},\zeta)-\lambda c(\xi,\zeta)}{\varepsilon}\right)\right].$$
- $$\inf_{\lambda\in\mathbb{R}_{+}}\lambda\varrho+\mathbb{E}_{\widehat{\xi}\sim\widehat{\mathbb{P}}_{N}}[\varphi_{\varepsilon}(\mathbf{x},\lambda,\widehat{\xi})]=\inf_{\lambda\in[0,\lambda_{\max}]}\lambda\varrho+\mathbb{E}_{\widehat{\xi}\sim\widehat{\mathbb{P}}_{N}}[\varphi_{\varepsilon}(\mathbf{x},\lambda,\widehat{\xi})].$$
- $$m_c=\max_{k\in\llbracket N\rrbracket}\mathbb{E}_{\zeta\sim\mathcal{N}(\widehat{\xi}_{k},\sigma^{2}\mathbf{I})}[c(\widehat{\xi}_{k},\zeta)],\qquad \lambda_{\max}=\frac{2\|f\|_{\infty}}{\varrho-m_c},\qquad \varrho>m_c.$$$$
- $$\widehat{\varphi}_{\varepsilon}(\theta,\lambda,\xi)=\varepsilon\log\left[\frac{1}{K}\sum_{j=1}^{K}\exp\left(\frac{\ell(g_\theta(\xi+\sigma u_j),y)-\lambda c(\xi,\xi+\sigma u_j)}{\varepsilon}\right)\right],\quad u_j\sim\mathcal{N}(0,I).$$

## Implementation notes

Integrate this at the minibatch loss, before the backward pass. For each training pair (xi_i,y_i), draw K perturbations u_ij from N(0,I) and form zeta_ij = xi_i + sigma u_ij. Compute per-sample losses L_ij = ell(g_theta(zeta_ij),y_i) and transport costs C_ij = c(xi_i,zeta_ij); use squared normalized Euclidean cost C_ij = ||u_ij||_2^2/d initially. Set scores S_ij = (L_ij - lambda C_ij)/epsilon, compute A_i = epsilon times (logsumexp_j(S_ij) - log K), and optimize the minibatch objective mean_i(A_i) + lambda rho. The softmax weights w_ij = softmax_j(S_ij) produce the gradient sum_j w_ij grad_theta L_ij. For a low-variance first implementation, detach w_ij before weighting gradients; then compare against full differentiation. Update theta with AdamW. Update lambda by projected gradient descent on [0,lambda_max], using the derivative rho - mean_ij(w_ij C_ij). Estimate m_c from the running batch mean of C_ij and choose a conservative finite loss bound if ||f||_infty is unavailable, such as a running high percentile. First test on CIFAR-10 with a small ResNet-18, K in {4,8}, sigma in {0.01,0.03,0.06}, and compare ERM, Gaussian augmentation, PGD training, and this loss at equal forward-pass count. Success means higher corrupted-CIFAR or adversarial accuracy at unchanged clean accuracy, together with smoother training and no unstable inner optimization.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built the entropic Wasserstein minibatch loss with Gaussian perturbations, normalized squared transport cost, and numerically stable log-sum-exp evaluation. The mechanism manifested: the softmax value stayed within the predicted epsilon*log(K) max-loss gap (worst observed ratio 0.920), the finite-difference lambda gradient matched the weighted-cost formula to 1.9e-10, and Monte Carlo relative standard deviation scaled as K^-1/2 (observed ratios 0.502, 0.248, and 0.125 for K=32,128,512 relative to K=8). In the tiny classifier comparison, entropic augmentation reached 0.866 perturbed accuracy versus 0.870 for Gaussian augmentation and 0.862 for ERM, so the mechanism is real but no clear performance win was established.

### Mechanism check

- Verdict: Built the entropic Wasserstein minibatch loss with Gaussian perturbations, normalized squared transport cost, and numerically stable log-sum-exp evaluation. The mechanism manifested: the softmax value stayed within the predicted epsilon*log(K) max-loss gap (worst observed ratio 0.920), the finite-difference lambda gradient matched the weighted-cost formula to 1.9e-10, and Monte Carlo relative standard deviation scaled as K^-1/2 (observed ratios 0.502, 0.248, and 0.125 for K=32,128,512 relative to K=8). In the tiny classifier comparison, entropic augmentation reached 0.866 perturbed accuracy versus 0.870 for Gaussian augmentation and 0.862 for ERM, so the mechanism is real but no clear performance win was established.
- Confidence: 8/10
- Limitations: This is a small synthetic two-dimensional classification experiment, not CIFAR-10 or a ResNet. Lambda was fixed rather than jointly projected-updated, full differentiation through the softmax was used, no PGD baseline or corruption suite was tested, and the comparison used a single fixed seed and short training budget.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.4847
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_entropic.py](https://synthcore.org/code/143/bench_entropic.py)
- [bench_report.json](https://synthcore.org/code/143/bench_report.json)
- [bench_rerun_output.json](https://synthcore.org/code/143/bench_rerun_output.json)
- [experiment.py](https://synthcore.org/code/143/experiment.py)
- [report.md](https://synthcore.org/code/143/report.md)
- [report_bench_2026-08-30T173954.md](https://synthcore.org/code/143/report_bench_2026-08-30T173954.md)
- [results.json](https://synthcore.org/code/143/results.json)
- [Download all files as ZIP](https://synthcore.org/download/143)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.