# Kurtosis-calibrated gradient clipping - ID: 426 - Canonical URL: https://synthcore.org/idea/426/kurtosis-calibrated-gradient-clipping - API JSON: https://synthcore.org/api/idea/426.json - API Markdown: https://synthcore.org/api/idea/426.md - Verification status: beats_baseline - Source: [arXiv:2607.05226](https://arxiv.org/abs/2607.05226) - Category: regularization - Solves: stability, sample-efficiency - ML areas: optimizer, training-dynamics, regularization - Math tags: probability, statistics, optimization - Ratings: usefulness 6/10; difficulty 4/10; novelty 5/10 ## Idea description Choose gradient clipping thresholds from an explicit worst-case tail probability implied by an observed kurtosis bound, rather than using a fixed norm threshold or an empirical percentile. For a standardized centered gradient coordinate, the threshold achieving target outlier probability \(\delta\) is obtained by analytically inverting the paper's sharp tail formula. ## Mathematical statement The paper studies \(\mathcal{C}(\kappa)=\{X:\mathbb{E}X=0,\ \mathbb{E}X^2=1,\ \mathbb{E}X^4\leq\kappa\}\), where \(\kappa\geq1\) is a kurtosis upper bound and the third moment is unconstrained. It defines \(V_1(t,\kappa)=\sup_{X\in\mathcal{C}(\kappa)}\mathbb{P}(X\geq t)\). In the tail regime \(t\geq c(\kappa)\), the exact worst-case bound is \(V_1(t,\kappa)=(\kappa-1)/((t^2-1)^2+\kappa-1)\). The transition threshold is the larger positive solution of equality with Cantelli's bound, satisfying \(c(\kappa)^4-(\kappa+1)c(\kappa)^2+1=0\), and therefore \(c(\kappa)^2=((\kappa+1)+\sqrt{(\kappa+1)^2-4})/2\) for \(\kappa>1\). Inverting the tail formula for target probability \(\delta\in(0,1)\) gives \(t_\delta^2=1+\sqrt{(\kappa-1)(1/\delta-1)}\). For a scalar gradient coordinate \(g\), estimate mean \(\mu\), standard deviation \(\sigma>0\), and kurtosis \(\kappa=\mathbb{E}[(g-\mu)^4]/\sigma^4\), then clip at \(\mu+t_\delta\sigma\). The guarantee is distribution-free only when \(\kappa\) is a valid upper bound, so empirical estimates require a safety factor. ## Key formulas - $$\mathcal{C}(\kappa)=\bigl\{X\text{ real-valued}:\mathbb{E}X=0,\ \mathbb{E}X^2=1,\ \mathbb{E}X^4\leq\kappa\bigr\},\qquad V_1(t,\kappa)=\sup_{X\in\mathcal{C}(\kappa)}\mathbb{P}(X\geq t).$$ - $$V_1(t,\kappa)=\frac{\kappa-1}{(t^2-1)^2+\kappa-1},\qquad t\geq c(\kappa).$$ - $$c(\kappa)^4-(\kappa+1)c(\kappa)^2+1=0,\qquad c(\kappa)^2=\frac{(\kappa+1)+\sqrt{(\kappa+1)^2-4}}{2}.$$ - $$\delta=\frac{\kappa-1}{(t_\delta^2-1)^2+\kappa-1}\quad\Longrightarrow\quad t_\delta=\sqrt{1+\sqrt{(\kappa-1)\left(\frac{1}{\delta}-1\right)}}.$$ ## Implementation notes Integrate this into the per-layer optimizer immediately before the AdamW or SGD update, operating on each parameter tensor's gradient values. Maintain exponential moving averages of the batch gradient mean, variance, and fourth central moment: \(\mu\leftarrow\beta\mu+(1-\beta)\operatorname{mean}(g)\), \(v\leftarrow\beta v+(1-\beta)\operatorname{mean}((g-\mu)^2)\), and \(q\leftarrow\beta q+(1-\beta)\operatorname{mean}((g-\mu)^4)\). In practice use the current batch mean when computing central moments, because using the lagged \(\mu\) can inflate \(q\). Compute \(\hat\kappa=\max(1,q/(v^2+\epsilon))\), multiply it by a safety factor such as 1.5, and optionally cap it at a fixed maximum to prevent a single noisy batch from producing an unusably large threshold. For a target tail rate \(\delta\), calculate \(t_\delta\) from the displayed inversion. If \(t_\delta