△ Mechanism confirmed, baseline not beaten 2026

Kurtosis-robust contraction step controller

Implementation & benchmark of arXiv:2607.11982 — Contraction Certification from Streaming Data: Wasserstein Robustness and Compositional Stability for Interconnected Nonlinear System

Usefulness8/10

Difficulty5/10

Novelty7/10

Dynamics Stability Generalization Optimizer Training dynamics Scheduler Control theory Dynamical systems Probability Statistics Spectral theory

Source paper: Contraction Certification from Streaming Data: Wasserstein Robustness and Compositional Stability for Interconnected Nonlinear System arXiv:2607.11982 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Treat one optimizer update as a stochastic dynamical map and estimate its local contraction margin from recent parameter-update or gradient residuals. Reduce the usable margin, and therefore the learning rate or trust-region radius, by a Wasserstein/heavy-tail penalty based on online excess kurtosis so distribution shifts cause graceful step-size shrinkage rather than sudden divergence.

Formulas

$$\theta_{k+1}=T_k(\theta_k;\xi_k),\qquad \hat\beta_k=1-\|J_k\|_2,\qquad J_k=\frac{\partial T_k}{\partial\theta}.$$

$$\hat\kappa_k=\frac{\frac{1}{W}\sum_{s=1}^{W}(r_s-\bar r)^4}{\left[\frac{1}{W}\sum_{s=1}^{W}(r_s-\bar r)^2\right]^2}-3,\qquad \varepsilon_k=\max(0,\hat\kappa_k).$$

$$\beta_{\mathrm{cert},k}=\hat\beta_k-\rho_k(1+2\varepsilon_k),\qquad \eta_{k+1}=\eta_k\min\left(1,\frac{\max(0,\beta_{\mathrm{cert},k})}{\beta_0}\right).$$

Mathematical statement

Let $\theta_{k+1}=T_k(\theta_k;\xi_k)$ be a stochastic optimization map, let $J_k=\partial T_k/\partial\theta$ be its local Jacobian, and define the nominal contraction margin $\hat\beta_k=1-\|J_k\|_2$, estimated using power iteration or Jacobian-vector products. Let $r_s$ be recent gradient or update residuals and $\hat\kappa_k=\mathbb{E}[r_s^4]/\mathbb{E}[r_s^2]^2-3$ their empirical excess kurtosis. Set $\varepsilon_k=\max(0,\hat\kappa_k)$, let $\rho_k\geq0$ be a Wasserstein uncertainty radius, and use $\beta_{\mathrm{cert},k}=\hat\beta_k-\rho_k(1+2\varepsilon_k)$. Permit the update only when this certificate is positive, or scale the learning rate by $\min(1,\beta_{\mathrm{cert},k}/\beta_0)$, where $\beta_0>0$ is a target margin. The predicted instability boundary is $\hat\beta_k=\rho_k(1+2\varepsilon_k)$.

Implementation notes

Integrate this into AdamW or SGD immediately before applying the parameter update. Maintain a window of the last W per-parameter or layer-aggregated residuals; a practical residual is $r_k=\|g_k-g_{k-1}\|_2/(\|g_{k-1}\|_2+\delta)$. Estimate second and fourth moments with exponential moving averages rather than storing the complete window. Estimate $\|J_k\|_2$ with one or two power iterations through the optimizer-update map: perturb parameters by a vector $v$, recompute the proposed update, and divide the update difference by $\|v\|$. For large models, use JVPs on selected layers. Initialize $\rho_k$ as a calibrated multiple of the running standard deviation of the contraction estimate; this is an empirical proxy, not a formal confidence bound. Pseudocode: compute gradient; form proposed optimizer map $T_k$; estimate $\hat\beta_k$; update residual moments and $\hat\kappa_k$; compute $\beta_{\mathrm{cert},k}$; set $\eta_{\mathrm{eff}}=\eta\,\mathrm{clip}(\beta_{\mathrm{cert},k}/\beta_0,0,1)$; apply the optimizer update using $\eta_{\mathrm{eff}}$. First test a 2-4 layer MLP on MNIST with SGD and AdamW under Gaussian, Laplace, and 1% spike-contaminated minibatch gradients. Compare fixed learning rate and gradient clipping. The predicted signature is a stability boundary satisfying $\hat\beta\approx\rho(1+2\varepsilon)$, with effective learning rate decreasing approximately linearly in positive excess kurtosis.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built and numerically verified the kurtosis-robust contraction controller. The predicted affine shrinkage with positive excess kurtosis and the certificate boundary matched the toy sweeps, but the controller did not improve final loss over fixed SGD in the contaminated-gradient MLP.

Agent confidence: 8/10

Baseline: Fixed SGD: final full MSE 0.05566; zero divergence in the tested toy dynamics and MLP run.
Idea: Controller: final full MSE 0.06652; zero divergence. Toy scales matched exactly: Gaussian 0.8857, Laplace 0.1477, and 1% spike 0; predicted and observed kurtosis boundary were both 3.875.

Limitations: The contraction margin was a supplied scalar proxy rather than an actual JVP or power-iteration estimate. The test used a tiny regression MLP, SGD rather than AdamW, one seed, a short residual window, and no broad hyperparameter or formal Wasserstein-confidence evaluation.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.11982) Sep 2, 2026

No significant difference vs. the tuned baseline (+0.1%, p=1, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0009
Idea mean: 0.0009
Effect (Δ): 0 (+0.1%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 1 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 4 configs

Limitations:

Tested only the built-in dynamics track, rnn_small model, 20 epochs, 400 training and 200 test samples, Adam rather than SGD/AdamW variants, and a small hyperparameter grid. The local contraction estimate is an empirical update-gain proxy rather than a true JVP/power-iteration Jacobian norm; no robustness stress test with injected benchmark gradient spikes was run.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented the kurtosis-robust contraction controller as an Adam update intervention on the structurally matched dynamics track with the shared rnn_small architecture. Across 8 paired seeds, the idea had test MSE 0.00092645 versus the tuned baseline 0.00092547, with delta +0.000000984, 0 wins, and permutation p=1.0; therefore there was no benchmark win. The trained-model mechanism signature was confirmed: mean excess kurtosis was 3.021 and the observed mean effective scale was 0.9988, below 1 as predicted, but the controller was too weak to produce a meaningful metric change.

Baseline: dynamics/rnn_small Adam sweep; best lr=0.003, weight_decay=0.0; 8-seed test MSE mean=0.0009254663, std=0.0001773996
Idea: Kurtosis controller at best idea setting lr=0.003, rho_mult=0.03, beta0=0.50; 8-seed test MSE mean=0.0009264500, std=0.0001774783; paired delta=+0.0000009836, p=1.0, idea_wins=0/8

Benchmark result