Cameron–Martin Adversarial Training

Integrate this at the input or hidden-state sequence of a recurrent, state-space, or transformer model. Given a minibatch of sequences $x_{1:T}$, construct a temporal covariance matrix $K$; start with an RBF kernel $K_{ij}=\exp[-(i-j)^2/(2\ell^2)]$ plus $10^{-4}I$ for numerical stability. Maintain an adversarial perturbation $z\in\mathbb{R}^{T\times d}$, initialized to zero or sampled from the top eigenvectors of $K\otimes I_d$. For $J$ inner steps, compute $g=\nabla_z L_\theta(x+z)$ and update $z\leftarrow z+\eta_z[g-\beta(K^{-1}\otimes I_d)z]$; the second term is exactly the gradient of the Cameron–Martin energy. Optionally parameterize $z=(K^{1/2}\otimes I_d)u$ and optimize $u$ instead, which avoids explicit inversion and makes the penalty $\|u\|_2^2$. Then update model parameters using $\nabla_\theta L_\theta(x+z)$, with stop-gradient through the adversary for a first-order baseline. The paper supplies the Gaussian variational/game objective and the quadratic energy; estimate $K$ empirically from residual temporal noise, augmentation statistics, or a chosen kernel, and estimate $K^{1/2}$ using eigendecomposition or a truncated Lanczos approximation. The first cheap experiment is a small GRU or S4-style state-space model on sequential MNIST, HAR, or character-level Penn Treebank. Compare ordinary training, iid FGSM/PGD noise, and the proposed RKHS adversary at equal inner-loop FLOPs. Measure clean validation loss, loss under held-out correlated Gaussian perturbations, and worst-case loss versus perturbation energy. Success means lower correlated-noise loss and better clean generalization at the same parameter count, with especially large gains when the test perturbation correlation length differs from the training sequence statistics.