Online Effective-Ridge Correction
Implementation & benchmark of arXiv:2607.00207 — Homogenization of $\ell_2$-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent
Source paper: Homogenization of $\ell_2$-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent arXiv:2607.00207 ⓘ · analyzed Aug 29, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Track the implicit l2 regularization induced by adversarial SGD and explicitly correct it when the optimizer drifts toward an undesirable ridge strength. Apply the correction first to the final linear head or a low-dimensional adapter, where feature covariance and ridge estimates are tractable.
Formulas
Mathematical statement
The paper’s adversarial homogenized SDE shows that adversarial least-squares SGD has risk dynamics equivalent, up to dimension-free constants, to standard least-squares SGD with an adaptive learning rate and adaptive l2 regularization. At a convergent fixed point, the iterate has the ridge form w_infinity=(Sigma+lambda_eff I)^(-1)b, where Sigma=E[xx^T] is feature covariance, b=E[xy] is the feature-label cross-moment, I is the identity matrix, and lambda_eff>=0 is the effective regularization induced by SGD and adversarial training. Estimate Sigma and b with exponential moving averages. Fit the scalar lambda_hat_t by minimizing the distance between the current iterate and the ridge solution. If lambda0 is the desired regularization selected on validation data, add rho(lambda_hat_t-lambda0)w_t to the gradient. The correction is gated or smoothed when the fixed-point approximation is not yet reliable.
Implementation notes
Integrate this into the final linear classifier head or a low-dimensional adapter. Let H_t in R^(Bxd) be the penultimate features, y_t the labels, and w_t the head weights. After each adversarial minibatch, update Sigma_hat=alpha*Sigma_hat+(1-alpha)*H_t^T H_t/B and b_hat=alpha*b_hat+(1-alpha)*H_t^T y_t/B. Every K optimizer steps, estimate lambda_hat by evaluating the ridge path w(lambda)=(Sigma_hat+lambda I)^(-1)b_hat on a logarithmic grid over nonnegative lambda and refining the best interval with one-dimensional search. Smooth lambda_hat with a median or exponential filter. Modify the head gradient to g_corr=g+rho*(lambda_hat-lambda0)*w, where lambda0 is chosen from clean-versus-robust validation performance and rho is small. Use a diagonal Sigma approximation when d is large. Freeze the correction until covariance EMAs stabilize, and gate it if lambda_hat changes by more than a preset factor. The paper supplies the fixed-point ridge structure; alpha, K, rho, lambda0, and the gating rule are empirical. First test a linear and two-layer MLP head on CIFAR-10 or a Gaussian-mixture benchmark under l2-PGD. Compare adversarial SGD, fixed weight decay, and this controller. Success means lower seed-to-seed variance, fewer late robust-loss increases, and improved robust accuracy at matched clean accuracy.
Verification
Stage 1 · Toy mechanism gate: Failed ✗
Stage 2 · Mechanism transferred to benchmark: Not tested
Stage 2 · Practical benchmark result: Not run
Methodology: Toy-system gate first; the benchmark stage runs only after a pass. How verification works
Stage 1 — Mechanism check agent confidence 8/10
Built an online effective-ridge controller with covariance and cross-moment EMAs, logarithmic ridge-path fitting, smoothing, gating, and head-gradient correction. The math check recovered the planted lambda 0.37 as 0.3736 with 0.0036 absolute error, confirming the estimator when the iterate is at a ridge fixed point. However, on the five-seed adversarial Gaussian regression test, controller robust loss was 0.047963 versus 0.047970 for adversarial SGD, while robust-loss seed variance was slightly worse (0.000648 versus 0.000635); this is effectively no win.
- Agent confidence
- 8/10
- Baseline
- Adversarial SGD: mean clean loss 0.011746, robust loss 0.047970, robust-loss std 0.000635, weight norm 0.9636. Fixed weight decay: clean loss 0.012527, robust loss 0.048304, robust-loss std 0.000573.
- Idea
- Online controller: mean clean loss 0.011751, robust loss 0.047963, robust-loss std 0.000648, weight norm 0.9633; final fitted lambda values across seeds were approximately 0.0288-0.0375.
Limitations: Only a small linear Gaussian regression benchmark was tested, not CIFAR-10 or a two-layer MLP. The experiment used one adversarial radius, one controller hyperparameter setting, five seeds, 500 optimization steps, and did not measure late robust-loss increases, matched clean accuracy, or computational overhead. The planted-ridge sanity check assumes the iterate is exactly at a ridge solution, so it does not validate fixed-point approximation error during realistic transient training.
How to run: python3 online_ridge.py
Verdict computed by deterministic test code from paired-seed statistics — not by the language model.
Artifacts
- online_ridge.py 4.6 KB View Raw
- report.md 1.7 KB View
- results.json 3.9 KB View Raw