{
 "artifacts": null,
 "category": "optimization",
 "description": "Use a Polyak stepsize computed from the adversarial minibatch loss instead of a fixed learning rate or an inner exact line search. The paper’s dynamics indicate that adversarial SGD has state-dependent curvature and noise, and that Polyak control can avoid the non-monotone behavior caused by anisotropic covariance and ridge mismatch.",
 "formulas_latex": [
  "$$\\ell_\\epsilon(w;x,y)=\\max_{\\|\\delta\\|_2\\leq\\epsilon}(y-w^\\top(x+\\delta))^2=(|y-w^\\top x|+\\epsilon\\|w\\|_2)^2.$$",
  "$$F_t(w)=\\frac{1}{B}\\sum_{i=1}^{B}\\ell_\\epsilon(w;x_i,y_i),\\qquad g_t=\\nabla F_t(w_t).$$",
  "$$\\eta_t=\\operatorname{clip}\\!\\left(\\frac{F_t(w_t)-\\widehat F_{\\star,t}}{\\|g_t\\|_2^2+\\tau},\\eta_{\\min},\\eta_{\\max}\\right),\\qquad w_{t+1}=w_t-\\eta_t g_t.$$",
  "$$\\widehat F_{\\star,t}=\\min(\\widehat F_{\\star,t-1},\\operatorname{EMA}_\\beta[F_t])-c\\,\\operatorname{SE}_B(F_t).$$"
 ],
 "id": 300,
 "implementation": "Integrate this at the optimizer-update stage, after adversarial examples have been generated and before the parameter update. For each minibatch, generate x_adv with the existing PGD or single-step l2 attack, compute the robust loss F_t and gradient g_t, and retain per-example losses to estimate the standard error. Maintain loss_ema=beta*loss_ema+(1-beta)*F_t and set F_star_hat=min(F_star_hat,loss_ema)-c*SE, where SE is the batch standard deviation divided by sqrt(B). Compute eta=(loss_ema-F_star_hat)/(squared_norm(g)+tau), clamp it between eta_min and eta_max, and update parameters with theta=theta-eta*g. First evaluate pure Polyak SGD; then test Polyak scaling on Adam’s normalized direction d=m_hat/(sqrt(v_hat)+eps), using eta=(F-F_star_hat)/(g dot d+tau). The mathematical ingredients are the robust objective and Polyak ratio; beta, c, tau, and clipping limits are empirical. Use a small MLP and linear classifier on MNIST or CIFAR-10 with l2-PGD. Compare fixed-rate SGD, AdamW, and exact backtracking at equal gradient evaluations. Success means fewer robust-loss oscillations, faster robust-loss reduction at equal FLOPs, and equal or higher final PGD accuracy across seeds.",
 "math_summary": "For parameters w, sample (x,y), adversarial radius epsilon, and linear predictor f_w(x)=w^T x, define the residual r=y-w^T x. Maximizing squared error over an l2 perturbation gives the exact robust loss ell_epsilon(w;x,y)=(|r|+epsilon||w||_2)^2. For a minibatch of size B, F_t(w)=B^{-1} sum_i ell_epsilon(w;x_i,y_i), and g_t=gradient F_t(w_t). The Polyak update uses eta_t=(F_t(w_t)-F_star_hat,t)/(||g_t||_2^2+tau), where F_star_hat,t is a conservative estimate of the optimal robust loss and tau\u003e0 prevents division by zero. Because the true optimum is unknown for neural networks, maintain F_star_hat,t as the minimum observed exponential-moving-average loss minus a safety margin proportional to minibatch standard error. Clip eta_t to an empirically chosen interval.",
 "math_tags": [
  "optimization",
  "dynamical-systems",
  "probability"
 ],
 "ml_areas": [
  "optimizer",
  "training"
 ],
 "paper": {
  "arxiv_id": "2607.00207",
  "arxiv_url": "https://arxiv.org/abs/2607.00207",
  "summary_what_math_gives_to_ml": "The paper’s transferable asset is its high-dimensional dynamical picture of adversarial SGD: robust perturbations alter both the effective step size and the implicit l2 regularization, so ordinary constant-step SGD can oscillate or move non-monotonically even on least-squares problems. Its analysis identifies Polyak stepsizes as more reliable than exact line search when feature covariance is anisotropic or the effective ridge parameter is mismatched. A practical transfer is to replace fixed learning rates in adversarial training with a clipped robust-loss Polyak controller, optionally augmented by online estimation of the effective ridge strength. The first experiments should target adversarial linear probes and small CNNs, measuring robust-loss descent and final robustness at equal optimizer evaluations.",
  "title": "Homogenization of $\\ell_2$-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 3,
  "novelty": 5,
  "usefulness": 8
 },
 "solves": [
  "stability",
  "accuracy",
  "sample-efficiency"
 ],
 "title": "Robust Polyak SGD",
 "url": "https://synthcore.org/idea/300/robust-polyak-sgd",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}