# Sharp Sub-Gamma Laplace Regularization

- ID: 3052
- Canonical URL: https://synthcore.org/idea/3052/sharp-sub-gamma-laplace-regularization
- API JSON: https://synthcore.org/api/idea/3052.json
- API Markdown: https://synthcore.org/api/idea/3052.md
- Verification status: unverified
- Source: [arXiv:2609.02398](https://arxiv.org/abs/2609.02398)
- Category: regularization
- Solves: stability, generalization, accuracy
- ML areas: regularization, loss, training
- Math tags: probability, convex-analysis, statistics, optimization
- Ratings: usefulness 5/10; difficulty 6/10; novelty 6/10

## Idea description

Replace ad hoc Gaussian or Laplace noise injection with a Laplace majorant calibrated to the observed finite-range sub-Gamma parameters of a neural perturbation. For convex perturbation losses, the calibration guarantees that the expected loss under the scaled Laplace noise upper-bounds the expected loss under every centered random perturbation satisfying the same Bernstein-type MGF constraint.

## Mathematical statement

The paper studies convex order: a centered random variable X is dominated by cL if E[f(X)] <= E[f(cL)] for every convex f, where L is a centered unit-scale Laplace random variable. The sub-Gamma constraint to implement is the Bernstein MGF condition E[exp(lambda X)] <= exp(sigma^2 lambda^2/(2(1-alpha |lambda|))) for |lambda| < 1/alpha, where sigma >= 0 is the variance proxy and alpha >= 0 is the scale parameter; apply the condition separately to positive and negative lambda. Define the sharp scale c_star(sigma, alpha) as the smallest c such that E[(X-t)_+] <= E[(cL-t)_+] for every threshold t and every centered X satisfying the MGF constraint. The hinge-function condition is sufficient because convex functions on the real line are affine functions plus nonnegative mixtures of hinges. The paper states that c_star is strictly larger than sigma vee alpha for the sub-Gamma class and that the optimum is attained by an asymmetric two-point distribution, so c_star can be estimated by optimizing over X taking values x_- < 0 and x_+ > 0 with probabilities p and 1-p, subject to p x_- + (1-p)x_+ = 0 and the finite-range MGF inequalities. For L with density p_L(l)=0.5 exp(-|l|), E[exp(lambda cL)] = 1/(1-c^2 lambda^2) for |lambda| < 1/c, and E[(cL-t)_+] has a closed form.

## Key formulas

- $$\mathbb{E}[f(X)]\leq\mathbb{E}[f(cG)]\qquad\text{for every convex }f\colon\mathbb{R}\to\mathbb{R}.$$
- $$\mathbb{E}[e^{\lambda X}]\leq\exp\!\left(\frac{\sigma^{2}\lambda^{2}}{2(1-\alpha|\lambda|)}\right),\qquad |\lambda|<\alpha^{-1}.$$
- $$c_\star=\inf\left\{c>0:\ \sup_{X\in\mathcal{S}(\sigma,\alpha)}\mathbb{E}[(X-t)_+]\leq\mathbb{E}[(cL-t)_+]\ \text{for every }t\in\mathbb{R}\right\}.$$
- $$\mathbb{E}[(cL-t)_+]=\begin{cases}\frac{c}{2}e^{-t/c},&t\geq0,\\-t+\frac{c}{2}e^{t/c},&t<0,\end{cases}\qquad \mathbb{E}[e^{\lambda cL}]=\frac{1}{1-c^2\lambda^2},\ |\lambda|<c^{-1}.$$

## Implementation notes

Integrate this at a scalar perturbation point, initially the pre-logit vector of a classifier or one selected hidden activation channel. During a calibration pass, collect centered scalar residuals X from minibatches and estimate sigma and alpha by fitting the empirical MGF envelope over a grid of lambda values, using a safety multiplier such as 1.1. Compute c_star offline by constrained optimization over asymmetric two-point laws. Parameterize p in (0,1), set x_-=-r and x_+=pr/(1-p) to enforce zero mean, and minimize c subject to log[p exp(lambda x_-)+(1-p)exp(lambda x_+)] <= sigma^2 lambda^2/[2(1-alpha lambda)] and the analogous inequality at negative lambda for grid points satisfying |lambda|<1/alpha. For each candidate c, check the hinge inequalities over a threshold grid t_k using the displayed closed form for the Laplace hinge expectation, then binary-search c. In training, sample L~Laplace(0,1) and add delta=c_star*sigma*L to the chosen scalar or normalized channel before the loss computation. If the loss is not convex in the perturbation, use a local quadratic surrogate with gradient g and curvature h estimated by finite differences. Pseudocode: estimate (sigma,alpha); solve c_star; for each batch compute z; sample delta; evaluate loss(z+delta); backpropagate. Compare no noise, Gaussian noise with standard deviation sigma, and uncalibrated Laplace noise with scale sigma on CIFAR-10 using a two-layer MLP and a linear softmax head. Success means lower worst-seed validation loss or improved clean accuracy at equal noise level, reduced gradient-norm variance, and empirical confirmation that held-out convex hinge or quadratic losses under measured perturbations are below the scaled-Laplace estimate.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.