# Kurtosis-robust contraction step controller

- ID: 775
- Canonical URL: https://synthcore.org/idea/775/kurtosis-robust-contraction-step-controller
- API JSON: https://synthcore.org/api/idea/775.json
- API Markdown: https://synthcore.org/api/idea/775.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.11982](https://arxiv.org/abs/2607.11982)
- Category: dynamics
- Solves: stability, generalization
- ML areas: optimizer, training-dynamics, scheduler
- Math tags: control-theory, dynamical-systems, probability, statistics, spectral-theory
- Ratings: usefulness 8/10; difficulty 5/10; novelty 7/10

## Idea description

Treat one optimizer update as a stochastic dynamical map and estimate its local contraction margin from recent parameter-update or gradient residuals. Reduce the usable margin, and therefore the learning rate or trust-region radius, by a Wasserstein/heavy-tail penalty based on online excess kurtosis so distribution shifts cause graceful step-size shrinkage rather than sudden divergence.

## Mathematical statement

Let \(\theta_{k+1}=T_k(\theta_k;\xi_k)\) be a stochastic optimization map, let \(J_k=\partial T_k/\partial\theta\) be its local Jacobian, and define the nominal contraction margin \(\hat\beta_k=1-\|J_k\|_2\), estimated using power iteration or Jacobian-vector products. Let \(r_s\) be recent gradient or update residuals and \(\hat\kappa_k=\mathbb{E}[r_s^4]/\mathbb{E}[r_s^2]^2-3\) their empirical excess kurtosis. Set \(\varepsilon_k=\max(0,\hat\kappa_k)\), let \(\rho_k\geq0\) be a Wasserstein uncertainty radius, and use \(\beta_{\mathrm{cert},k}=\hat\beta_k-\rho_k(1+2\varepsilon_k)\). Permit the update only when this certificate is positive, or scale the learning rate by \(\min(1,\beta_{\mathrm{cert},k}/\beta_0)\), where \(\beta_0>0\) is a target margin. The predicted instability boundary is \(\hat\beta_k=\rho_k(1+2\varepsilon_k)\).

## Key formulas

- $$\theta_{k+1}=T_k(\theta_k;\xi_k),\qquad \hat\beta_k=1-\|J_k\|_2,\qquad J_k=\frac{\partial T_k}{\partial\theta}.$$
- $$\hat\kappa_k=\frac{\frac{1}{W}\sum_{s=1}^{W}(r_s-\bar r)^4}{\left[\frac{1}{W}\sum_{s=1}^{W}(r_s-\bar r)^2\right]^2}-3,\qquad \varepsilon_k=\max(0,\hat\kappa_k).$$
- $$\beta_{\mathrm{cert},k}=\hat\beta_k-\rho_k(1+2\varepsilon_k),\qquad \eta_{k+1}=\eta_k\min\left(1,\frac{\max(0,\beta_{\mathrm{cert},k})}{\beta_0}\right).$$

## Implementation notes

Integrate this into AdamW or SGD immediately before applying the parameter update. Maintain a window of the last W per-parameter or layer-aggregated residuals; a practical residual is \(r_k=\|g_k-g_{k-1}\|_2/(\|g_{k-1}\|_2+\delta)\). Estimate second and fourth moments with exponential moving averages rather than storing the complete window. Estimate \(\|J_k\|_2\) with one or two power iterations through the optimizer-update map: perturb parameters by a vector \(v\), recompute the proposed update, and divide the update difference by \(\|v\|\). For large models, use JVPs on selected layers. Initialize \(\rho_k\) as a calibrated multiple of the running standard deviation of the contraction estimate; this is an empirical proxy, not a formal confidence bound. Pseudocode: compute gradient; form proposed optimizer map \(T_k\); estimate \(\hat\beta_k\); update residual moments and \(\hat\kappa_k\); compute \(\beta_{\mathrm{cert},k}\); set \(\eta_{\mathrm{eff}}=\eta\,\mathrm{clip}(\beta_{\mathrm{cert},k}/\beta_0,0,1)\); apply the optimizer update using \(\eta_{\mathrm{eff}}\). First test a 2-4 layer MLP on MNIST with SGD and AdamW under Gaussian, Laplace, and 1% spike-contaminated minibatch gradients. Compare fixed learning rate and gradient clipping. The predicted signature is a stability boundary satisfying \(\hat\beta\approx\rho(1+2\varepsilon)\), with effective learning rate decreasing approximately linearly in positive excess kurtosis.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built and numerically verified the kurtosis-robust contraction controller. The predicted affine shrinkage with positive excess kurtosis and the certificate boundary matched the toy sweeps, but the controller did not improve final loss over fixed SGD in the contaminated-gradient MLP.

### Mechanism check

- Verdict: Built and numerically verified the kurtosis-robust contraction controller. The predicted affine shrinkage with positive excess kurtosis and the certificate boundary matched the toy sweeps, but the controller did not improve final loss over fixed SGD in the contaminated-gradient MLP.
- Confidence: 8/10
- Limitations: The contraction margin was a supplied scalar proxy rather than an actual JVP or power-iteration estimate. The test used a tiny regression MLP, SGD rather than AdamW, one seed, a short residual window, and no broad hyperparameter or formal Wasserstein-confidence evaluation.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 1
- Paired wins: 0/8
- Benchmark verdict: no measurable effect

## Artifacts

- [REPORT.md](https://synthcore.org/code/146/REPORT.md)
- [bench_experiment.py](https://synthcore.org/code/146/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/146/bench_report.json)
- [experiment.py](https://synthcore.org/code/146/experiment.py)
- [report.md](https://synthcore.org/code/146/report.md)
- [report_bench_2026-08-30T173927.md](https://synthcore.org/code/146/report_bench_2026-08-30T173927.md)
- [results.json](https://synthcore.org/code/146/results.json)
- [Download all files as ZIP](https://synthcore.org/download/146)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.