△ Mechanism confirmed, baseline not beaten 2026

Convex-gradient robust augmenter

Implementation & benchmark of arXiv:2608.22746 — Generative Neural Networks for Sinkhorn Distributionally Robust Hypothesis Testing

Usefulness6/10

Difficulty6/10

Novelty5/10

Training Accuracy Generalization Stability Training Data augmentation Regularization Generative Optimal transport Convex analysis Information theory Geometry

Source paper: Generative Neural Networks for Sinkhorn Distributionally Robust Hypothesis Testing arXiv:2608.22746 ⓘ · analyzed Aug 29, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace unconstrained adversarial example generation with an invertible transport map that is the gradient of a convex potential. For each class, the map pushes a kernel-smoothed empirical distribution toward a least-favorable distribution inside a prescribed KL/Sinkhorn ambiguity radius, producing hard but globally coherent training examples rather than pointwise perturbations.

Formulas

$$T_{\theta,k}(x)=\nabla_x u_{\theta,k}(x),\qquad u_{\theta,k}(\cdot)\ \text{convex},\qquad Q_{\theta,k}=T_{\theta,k\#}\bar P_k.$$

$$D_{\mathrm{KL}}(Q_{\theta,k}\|\bar P_k)=\mathbb{E}_{x\sim\bar P_k}\left[\log \bar p_k(x)-\log\left|\det J_{T_{\theta,k}}(x)\right|-\log \bar p_k\left(T_{\theta,k}(x)\right)\right].$$

$$\min_w\max_{\theta}\sum_{k\in\{0,1\}}\mathbb{E}_{x\sim\bar P_k}\left[\ell(f_w(T_{\theta,k}(x)),k)\right]-\lambda\sum_{k\in\{0,1\}}\max\left(0,D_{\mathrm{KL}}(Q_{\theta,k}\|\bar P_k)-\rho\right).$$

Mathematical statement

The paper uses kernel-smoothed reference distributions and Brenier's theorem: if a source measure $\bar P$ and target measure $Q$ are absolutely continuous, the optimal transport map has the form $T(x)=\nabla_x u(x)$, where $u$ is convex. For source samples $x\sim\bar P$, define $Q_\theta=T_{\theta\#}\bar P$, the pushforward distribution under $T_\theta$. The adaptation constrains the generated distribution using $D_{\mathrm{KL}}(Q_\theta\|\bar P)\leq\rho$, where $\rho\geq0$ is the robustness radius. When $T_\theta$ is invertible and $\bar p$ is the smoothed source density, the change-of-variables identity gives $D_{\mathrm{KL}}(Q_\theta\|\bar P)=\mathbb E_{x\sim\bar P}[\log\bar p(x)-\log|\det J_{T_\theta}(x)|-\log\bar p(T_\theta(x))]$, with $J_T$ the Jacobian. Train the classifier against the worst transported class distributions using $\min_w\max_\theta \sum_k\mathbb E_{x\sim\bar P_k}[\ell(f_w(T_{\theta,k}(x)),k)]-\lambda\max(0,D_{\mathrm{KL}}(Q_{\theta,k}\|\bar P_k)-\rho)$. Convexity of $u_\theta$ is enforced by nonnegative input-to-hidden weights, making $T_\theta=\nabla u_\theta$ monotone and globally structured rather than an arbitrary perturbation field.

Implementation notes

(1) Integration point: add one class-conditional transport augmenter before the classifier or encoder. For each minibatch of class-k inputs x, compute a convex potential u_{theta,k}(x), then obtain x_adv = grad_x u_{theta,k}(x). Use a small ICNN/HyCNN-style potential with positive linear combinations of convex softplus or squared-ReLU features in x, and optionally condition parameters on the class label. Initialize u(x)=0.5||x||^2 so that the initial transport is the identity.

(2) Pseudocode:

for x,y in minibatch:
    x_s = x + Normal(0, sigma^2 I)
    u = convex_potential(theta[y], x_s)
    x_adv = grad_x(u.sum(), x_s)
    logp0 = log_kde(y, x_s.detach())
    logp1 = log_kde(y, x_adv)
    logdet = logdet_jacobian(theta[y], x_s)
    kl = mean(logp0 - logdet - logp1)
    robust_loss = CE(classifier(x_adv), y) - lambda*relu(kl-rho)
    update theta by ascent on robust_loss
    update classifier weights w by descent on robust_loss

Use 1-3 ascent steps for theta and one descent step for w; detach classifier gradients when updating theta only. (3) The mathematical components are the convex-gradient map, pushforward distribution, Jacobian change-of-variables KL estimate, and radius penalty. Estimate the smoothed source density $\bar p_k$ with a Gaussian KDE on the current training subset, or use a learned conditional density in high dimension. Compute $\log\det J_T$ exactly for low-dimensional inputs; for larger inputs use Hutchinson trace estimation or a triangular Jacobian parameterization. (4) First experiment: train a 2-layer MLP on two-moons and a small ResNet on binary Colored-MNIST or CIFAR-10. Compare ERM, Gaussian augmentation, PGD adversarial training, and unconstrained MLP augmenters at equal update and generated-sample counts. Sweep $\rho$, and report clean accuracy, held-out color/noise/rotation-shift accuracy, KL-budget violations, and transport displacement. Success means improved worst-shift accuracy without materially reducing clean accuracy, with empirical KL remaining close to the prescribed radius.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built an exact quadratic convex-gradient transport prototype and a two-moons comparison. The mechanism checks confirmed the predicted identity minimum, quadratic near-identity KL scaling, and linear dimension scaling; the Monte Carlo change-of-variables KL estimate matched the analytic value within 0.72%. The classifier result was not a clear practical win: ConvexTransport matched ERM on shifted accuracy but reduced clean accuracy.

Agent confidence: 9/10

Baseline: ERM: clean accuracy 0.9633, shifted accuracy 0.9633, clean log loss 0.0775; Gaussian: clean accuracy 0.9700, shifted accuracy 0.9600
Idea: ConvexTransport: clean accuracy 0.9533, shifted accuracy 0.9633, clean log loss 0.1099; prescribed KL radius rho=0.06137
Math checks: Identity KL minimum: exact zero at scale 1.0; quadratic KL ratio mean 0.99388 versus prediction 1.0; dimension slope 0.0376784 observed versus predicted 0.0376784; change-of-variables KL relative error 0.72%

Limitations: Only a quadratic isotropic/diagonal transport family and one small two-moons experiment were tested. Learned class-conditional ICNN/HyCNN potentials, Sinkhorn optimization, adversarial ascent, KDE estimation, PGD and unconstrained augmenter baselines, multiple seeds, image data, and learned-radius sweeps were not tested.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.22746) Sep 2, 2026

Ahead of the baseline (−0.7%) but not statistically significant (p=0.8087, wins 5 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0013
Idea mean: 0.0013
Effect (Δ): -0 (−0.7%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.8087 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±4.9%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 0 configs

Limitations:

Only the built-in controlled-pendulum dynamics track was tested. The implementation uses a low-dimensional isotropic quadratic convex potential rather than a learned ICNN, KDE/Sinkhorn ambiguity optimization, or class-conditional adversarial ascent; no vision or custom track evaluation was performed.

How to run: python3 bench_convex.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the convex-gradient transport augmenter and ran the full 8-seed paired dynamics benchmark with rnn_small, equal learning-rate search space, and a tuned Gaussian baseline sweep. The best idea setting was the identity transport and produced a negligible non-significant improvement (delta_mean=-9.39e-06, p=0.8087), so the promised practical win was not observed. The non-identity mechanism signature was confirmed: observed versus predicted displacement differed by 1.02e-06 relative error, but this does not establish a task-performance gain.

Baseline: Best Gaussian baseline: lr=0.003, noise=0.05; 8-seed MSE mean=0.0013185284, std=0.0004747611
Comparison: delta_mean=-0.00000939018, idea_wins=5/8, permutation p=0.8087, verdict=no measurable effect
Idea: Best convex-gradient setting: lr=0.003, scale=1.00; 8-seed MSE mean=0.0013091382, std=0.0004729142
Mechanism signature: Non-identity scale=1.04: predicted displacement=0.16550005, observed displacement=0.16549988, relative error=1.02e-06, confirmed=true

Benchmark result