# Hermite compound-Poisson feature noise

- ID: 2393
- Canonical URL: https://synthcore.org/idea/2393/hermite-compound-poisson-feature-noise
- API JSON: https://synthcore.org/api/idea/2393.json
- API Markdown: https://synthcore.org/api/idea/2393.md
- Verification status: unverified
- Source: [arXiv:2608.16272](https://arxiv.org/abs/2608.16272)
- Category: regularization
- Solves: generalization, stability
- ML areas: regularization, training, mlp
- Math tags: probability, stochastic-processes, statistics, approximation-theory
- Ratings: usefulness 5/10; difficulty 5/10; novelty 4/10

## Idea description

Replace ordinary additive or multiplicative activation noise with a nonnegative count-valued perturbation generated by the Hermite operator kernel. For a nonnegative feature x, sample an integer N whose distribution is exactly the operator's weight sequence and feed N/n to the next layer; the parameter alpha controls an additional even-jump component and therefore changes the noise geometry independently of the ordinary Poisson component.

## Mathematical statement

The paper defines the positive linear operator $$(G_{n,\alpha}f)(x)=e^{-(nx+\alpha x^{2})}\sum_{\nu=0}^{\infty}\frac{x^{\nu}}{\nu!}H_{\nu}(n,\alpha)f\left(\frac{\nu}{n}\right),$$ where $n\in\mathbb{N}$, $\alpha\ge 0$, and $x\ge 0$, with $$H_{\nu}(n,\alpha)=\nu!\sum_{j=0}^{\lfloor \nu/2\rfloor}\frac{n^{\nu-2j}\alpha^j}{(\nu-2j)!j!}.$$ The exponential generating function is $$\sum_{\nu=0}^{\infty}\frac{x^{\nu}}{\nu!}H_{\nu}(n,\alpha)=\exp(nx+\alpha x^2).$$ Normalizing the coefficients gives the probability generating function $$\sum_{\nu=0}^{\infty}p_\nu(x)z^\nu=\exp\left(nx(z-1)+\alpha x^2(z^2-1)\right),$$ so the random index is $N=P+2Q$, with independent $P\sim\operatorname{Poisson}(nx)$ and $Q\sim\operatorname{Poisson}(\alpha x^2)$. Therefore $G_{n,\alpha}f(x)=\mathbb{E}[f(N/n)]$, $\mathbb{E}[N/n]=x+2\alpha x^2/n$, and $\operatorname{Var}(N/n)=x/n+4\alpha x^2/n^2$. The mean-preserving adaptation $Y=(N-2\alpha x^2)/n$ satisfies $\mathbb{E}[Y]=x$ while retaining the same variance.

## Key formulas

- $$(G_{n,\alpha}f)(x)=e^{-(nx+\alpha x^{2})}\sum_{\nu=0}^{\infty}\frac{x^{\nu}}{\nu!}H_{\nu}(n,\alpha)f\left(\frac{\nu}{n}\right)$$
- $$H_{\nu}(n,\alpha)=\nu!\sum_{j=0}^{\lfloor \nu/2\rfloor}\frac{n^{\nu-2j}\alpha^j}{(\nu-2j)!j!}$$
- $$\sum_{\nu=0}^{\infty}p_\nu(x)z^\nu=\exp\left(nx(z-1)+\alpha x^2(z^2-1)\right),\qquad p_\nu(x)=e^{-(nx+\alpha x^2)}\frac{x^\nu}{\nu!}H_\nu(n,\alpha)$$
- $$N=P+2Q,\quad P\sim\operatorname{Poisson}(nx),\quad Q\sim\operatorname{Poisson}(\alpha x^2),\quad Y=\frac{N-2\alpha x^2}{n},\quad \operatorname{Var}(Y)=\frac{x}{n}+\frac{4\alpha x^2}{n^2}$$

## Implementation notes

Integrate the method at the input of one or two MLP blocks, after enforcing nonnegative features with Softplus or ReLU. For a feature tensor X, choose integer n and nonnegative alpha; sample elementwise P~Poisson(n*X) and Q~Poisson(alpha*X^2), set N=P+2*Q, and use Y=(N-2*alpha*X^2)/n during training. The subtraction makes the stochastic layer unbiased because E[Y]=X; optionally replace it with Y=N/n if a positive mean shift is desired. Pseudocode is: `X=softplus(h); P=Poisson(n*X); Q=Poisson(alpha*X*X); Y=(P+2*Q-2*alpha*X*X)/n; h_next=Linear(Y)`. The exact mathematical quantities are the compound-Poisson sampling law and its moments. Estimate alpha by validation search or learn `alpha=softplus(a)` per layer; use fixed n in {4,16,64}. Since Poisson sampling is not pathwise differentiable, start with a straight-through estimator: use the sampled Y in the forward pass and the derivative of the mean-preserving Gaussian proxy `X + sqrt(X/n+4*alpha*X^2/n^2)*epsilon` in the backward pass. A simpler ablation can use the deterministic expectation Y=X to verify the implementation. The first cheap experiment is a 3-layer MLP on MNIST or Fashion-MNIST, comparing this layer against dropout and Gaussian noise at matched activation variance. Sweep alpha in {0,0.25,1} and n in {4,16,64}; measure test accuracy, calibration error, gradient-norm variance, and train/test loss gap. Success means equal-or-better test accuracy with a smaller train/test gap or lower gradient variance at matched forward noise variance.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.