Homogenization of $\ell_2$-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent
arXiv:2607.00207
2026
Optimization
2 ideas extracted · analyzed Aug 29, 2026
What the math gives to ML
The paper’s transferable asset is its high-dimensional dynamical picture of adversarial SGD: robust perturbations alter both the effective step size and the implicit l2 regularization, so ordinary constant-step SGD can oscillate or move non-monotonically even on least-squares problems. Its analysis identifies Polyak stepsizes as more reliable than exact line search when feature covariance is anisotropic or the effective ridge parameter is mismatched. A practical transfer is to replace fixed learning rates in adversarial training with a clipped robust-loss Polyak controller, optionally augmented by online estimation of the effective ridge strength. The first experiments should target adversarial linear probes and small CNNs, measuring robust-loss descent and final robustness at equal optimizer evaluations.
Ideas from this paper
Unverified
Re-invented
2026
Use a Polyak stepsize computed from the adversarial minibatch loss instead of a fixed learning rate or an inner exact line search. The paper’s dynamics indicate that adversarial SGD has state-dependent curvature and noise, and that Polyak control can avoid the non-monotone behavior caused by anisotropic covariance and ridge mismatch.
Useful8/10
Difficulty3/10
Novelty5/10
Unverified
2026
Track the implicit l2 regularization induced by adversarial SGD and explicitly correct it when the optimizer drifts toward an undesirable ridge strength. Apply the correction first to the final linear head or a low-dimensional adapter, where feature covariance and ridge estimates are tractable.
Useful6/10
Difficulty5/10
Novelty6/10