# Shape-Optimized Private Gradient Noise

- ID: 2956
- Canonical URL: https://synthcore.org/idea/2956/shape-optimized-private-gradient-noise
- API JSON: https://synthcore.org/api/idea/2956.json
- API Markdown: https://synthcore.org/api/idea/2956.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.31138](https://arxiv.org/abs/2608.31138)
- Category: training
- Solves: accuracy, stability, sample-efficiency
- ML areas: optimizer, training, regularization
- Math tags: probability, statistics, optimization, information-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace fixed Gaussian noise in a private optimizer with generalized-Gaussian noise whose shape p is selected for the actual clipped-gradient sensitivity and privacy budget. For every candidate p, numerically find the minimum scale b satisfying the hockey-stick privacy constraint, then choose the p minimizing a gradient-update utility moment such as variance or expected absolute magnitude.

## Mathematical statement

The paper defines approximate differential privacy through the epsilon hockey-stick divergence D_epsilon(P||Q)=sup_A{P(A)-exp(epsilon)Q(A)} and requires D_epsilon(P_D||P_D')<=delta for neighboring datasets. Use the generalized-Gaussian density f_{p,b}(z)=p/[2b Gamma(1/p)] exp[-(|z|/b)^p] for p in [1,infinity) and scale b>0. For an additive scalar mechanism M(D)=q(D)+Z with sensitivity Delta=|q(D)-q(D')|, the privacy constraint is D_epsilon(f_{p,b}(.)||f_{p,b}(. - Delta))<=delta. The scale-homogeneous utility moment is E|Z|^m=b^m Gamma((m+1)/p)/Gamma(1/p), so minimizing utility for fixed p means using the smallest privacy-feasible b(p); shape selection minimizes U_m(p)=b(p)^m Gamma((m+1)/p)/Gamma(1/p). Since rescaling Delta rescales b(p) by the same factor, the optimal p should be invariant to the absolute sensitivity scale when the normalized sensitivity geometry is unchanged.

## Key formulas

- $$D_{\varepsilon}(P\|Q):=\sup_A\{P(A)-e^{\varepsilon}Q(A)\},\qquad D_{\varepsilon}(P_D\|P_{D'})\leq\delta\quad(D\sim D').$$
- $$f_{p,b}(z)=\frac{p}{2b\,\Gamma(1/p)}\exp\!\left[-\left(\frac{|z|}{b}\right)^p\right],\qquad p\in[1,\infty),\ b>0.$$
- $$D_{\varepsilon}(f_{p,b}\|f_{p,b,\Delta})=\int_{\mathbb R}\left[f_{p,b}(x)-e^{\varepsilon}f_{p,b}(x-\Delta)\right]_+dx\leq\delta,$$
- $$U_m(p)=b(p)^m\frac{\Gamma((m+1)/p)}{\Gamma(1/p)},\qquad p^*=\arg\min_{p\in[1,p_{\max}]}U_m(p).$$

## Implementation notes

Integrate this at the noise-injection line of a private optimizer, initially for a scalar clipped gradient query or one coordinate of a clipped gradient. Clip each per-example gradient to norm C, so use scalar sensitivity Delta=C or 2C according to the neighboring-dataset convention; keep that convention identical for all baselines. For every candidate p on a grid such as {1.0,1.25,...,8.0}, solve for b(p) by bisection. At each b, evaluate the hockey-stick integral numerically by constructing f(x) and f(x-Delta), identifying where f(x)-exp(epsilon)f(x-Delta)>0, and integrating the positive difference with adaptive quadrature over an interval initially covering 12b+Delta. Expand the interval until the omitted tail is below 1e-10. Choose the smallest b whose divergence is at most delta minus a numerical safety margin, then calculate U_m(p) and select p*. Sample noise using inverse-CDF sampling: for u uniform on [-1/2,1/2], set z=b sign(u)[-log(1-2|u|)]^(1/p). Apply g_private=g_clipped+z before the optimizer update. The paper supplies the divergence, distribution family, and moment objective; quadrature and bisection are numerical implementations. First test logistic regression and a small MNIST MLP, comparing p=1, p=2, and selected p under equal epsilon, delta, steps, clipping C, parameter count, and training FLOPs. Pre-register: measured divergence must be within 1% of delta at the selected scale; multiplying Delta by 0.5, 1, and 2 must leave p* unchanged within one grid step while b* scales proportionally within 3%; and at least one privacy regime must produce a 5% lower U_2 or U_1 than both Laplace and Gaussian. Fixing p=2 is the ablation. The transfer is falsified if calibration is non-conservative, p* changes substantially under sensitivity rescaling, or no moment/task improvement appears across privacy regimes.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built generalized-Gaussian sampling, hockey-stick divergence calibration, grid-based shape selection, scaling checks, and a clipped scalar mean-estimation experiment. Quadrature calibration reached delta accurately, Monte Carlo moments matched the analytic formula, and b scaled proportionally with sensitivity while p*=1 remained invariant. However, p=1 was optimal in all tested privacy regimes; the optimized arm is therefore Laplace and its small MSE difference is not evidence of a shape-optimization gain.

### Mechanism check

- Verdict: Built generalized-Gaussian sampling, hockey-stick divergence calibration, grid-based shape selection, scaling checks, and a clipped scalar mean-estimation experiment. Quadrature calibration reached delta accurately, Monte Carlo moments matched the analytic formula, and b scaled proportionally with sensitivity while p*=1 remained invariant. However, p=1 was optimal in all tested privacy regimes; the optimized arm is therefore Laplace and its small MSE difference is not evidence of a shape-optimization gain.
- Confidence: 9/10
- Limitations: Only a scalar clipped mean-estimation task was tested, not logistic regression or MNIST MLP. The task prototype uses a sensitivity-calibrated scalar mechanism but does not implement a full DP-SGD accountant or composition across iterations; no wall-clock/FLOP comparison was performed.

## Artifacts

- [report.md](https://synthcore.org/code/1149/report.md)
- [results.json](https://synthcore.org/code/1149/results.json)
- [shape_private_noise.py](https://synthcore.org/code/1149/shape_private_noise.py)
- [sweep_results.json](https://synthcore.org/code/1149/sweep_results.json)
- [verify_and_sweep.py](https://synthcore.org/code/1149/verify_and_sweep.py)
- [Download all files as ZIP](https://synthcore.org/download/1149)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.