Inverse-Gamma Kappa SGD

Integrate this at the optimizer-noise stage, after computing the minibatch gradient and before applying momentum or Adam preconditioning. A minimal implementation is: first compute gₜ; second draw uₜ from Gamma(α, 1) and set Sₜ = β divided by uₜ, which is an exact inverse-gamma sample; third draw εₜ from N(0, I); fourth form the perturbed gradient g̃ₜ = gₜ + σ√Sₜ εₜ; and fifth pass g̃ₜ to the chosen optimizer. To avoid dimension-dependent explosions, use per-layer normalized noise σₗ√Sₜ εₜ divided by √dₗ, or sample one Sₜ per layer. The paper supplies the inverse-gamma-to-kappa mechanism and its asymptotic marginal. The engineer should estimate the effective tail index from gradient increments using a Hill estimator or Student-t maximum likelihood. Start with a cheap experiment using a two-layer MLP on MNIST or CIFAR-10, with momentum SGD as the baseline, matched mean update norm, and κ values 1.75, 2.25, 3, 5, 10, and infinity. Measure loss, gradient-noise variance, fourth empirical moment, escape from sharp minima, and test accuracy. The quantitative prediction is a moment transition: the estimated variance should become unstable below κ = 3/2, while the empirical fourth moment should show a strong divergence or finite-sample cutoff transition near κ = 5/2. For large κ, training curves and update kurtosis should approach Gaussian-SGD values. For finite κ, update magnitudes should exhibit the tail law P(||Δθ|| > r) proportional to r raised to the power −(2κ − 1) over the scaling range. Record raw statistics before clipping and separately test whether clipping changes the transition.