# Inverse-Gamma Kappa SGD

- ID: 1764
- Canonical URL: https://synthcore.org/idea/1764/inverse-gamma-kappa-sgd
- API JSON: https://synthcore.org/api/idea/1764.json
- API Markdown: https://synthcore.org/api/idea/1764.md
- Verification status: unverified
- Source: [arXiv:2608.03960](https://arxiv.org/abs/2608.03960)
- Category: training
- Solves: stability, generalization, sample-efficiency
- ML areas: optimizer, training-dynamics, diffusion-sampling
- Math tags: statistical-mechanics, probability, stochastic-processes, optimization
- Ratings: usefulness 6/10; difficulty 3/10; novelty 6/10

## Idea description

Inject scale-mixture noise into SGD by sampling the perturbation magnitude from an inverse-gamma distribution rather than using fixed-variance Gaussian noise. The resulting gradient updates have kappa or Student-t tails, allowing rare large exploratory steps while retaining an explicit control parameter for the Gaussian limit and for the existence of noise moments.

## Mathematical statement

Let gₜ = ∇θ ℓ(θₜ; Bₜ) be the minibatch gradient, η the learning rate, εₜ distributed as a d-dimensional standard Gaussian, and Sₜ distributed as InvGamma(α, β) with density p(S) = βᵅ S⁻ᵅ⁻¹ exp(−β/S) divided by Γ(α), for S greater than zero. Use the update θₜ₊₁ = θₜ − η[gₜ + σ√Sₜ εₜ]. Integrating out Sₜ gives a multivariate Student-t distribution with 2α degrees of freedom. The paper's three-dimensional kappa marginal is proportional to [1 + v² / ((κ − 3/2)θ²)] raised to the power −(κ + 1). Its radial moment of order p exists only for p < 2κ − 1, so variance requires κ > 3/2 and fourth moments require κ > 5/2. As κ tends to infinity, the kappa law converges to the Maxwellian or Gaussian law.

## Key formulas

- P(v | κ, θ) = [π(κ − 3/2)θ²]⁻³ᐟ² Γ(κ + 1) / Γ(κ − 1/2) × [1 + v² / ((κ − 3/2)θ²)]⁻(κ + 1).
- p(S | α, β) = βᵅ S⁻ᵅ⁻¹ exp(−β/S) / Γ(α), with S > 0.
- θₜ₊₁ = θₜ − η[gₜ + σ√Sₜ εₜ], where εₜ is distributed as N(0, I_d) and Sₜ is distributed as InvGamma(α, β).
- E[||v||ᵖ] is finite if and only if p < 2κ − 1; κ tending to infinity implies convergence to the Gaussian Maxwellian distribution.

## Implementation notes

Integrate this at the optimizer-noise stage, after computing the minibatch gradient and before applying momentum or Adam preconditioning. A minimal implementation is: first compute gₜ; second draw uₜ from Gamma(α, 1) and set Sₜ = β divided by uₜ, which is an exact inverse-gamma sample; third draw εₜ from N(0, I); fourth form the perturbed gradient g̃ₜ = gₜ + σ√Sₜ εₜ; and fifth pass g̃ₜ to the chosen optimizer. To avoid dimension-dependent explosions, use per-layer normalized noise σₗ√Sₜ εₜ divided by √dₗ, or sample one Sₜ per layer. The paper supplies the inverse-gamma-to-kappa mechanism and its asymptotic marginal. The engineer should estimate the effective tail index from gradient increments using a Hill estimator or Student-t maximum likelihood. Start with a cheap experiment using a two-layer MLP on MNIST or CIFAR-10, with momentum SGD as the baseline, matched mean update norm, and κ values 1.75, 2.25, 3, 5, 10, and infinity. Measure loss, gradient-noise variance, fourth empirical moment, escape from sharp minima, and test accuracy. The quantitative prediction is a moment transition: the estimated variance should become unstable below κ = 3/2, while the empirical fourth moment should show a strong divergence or finite-sample cutoff transition near κ = 5/2. For large κ, training curves and update kurtosis should approach Gaussian-SGD values. For finite κ, update magnitudes should exhibit the tail law P(||Δθ|| > r) proportional to r raised to the power −(2κ − 1) over the scaling range. Record raw statistics before clipping and separately test whether clipping changes the transition.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.