# Marginal-Preserving Difference Noise

- ID: 438
- Canonical URL: https://synthcore.org/idea/438/marginal-preserving-difference-noise
- API JSON: https://synthcore.org/api/idea/438.json
- API Markdown: https://synthcore.org/api/idea/438.md
- Verification status: unverified
- Source: [arXiv:2607.05460](https://arxiv.org/abs/2607.05460)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: diffusion, vae, regularization, training-dynamics
- Math tags: probability, measure-theory, stochastic-processes
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Construct two latent variables X and Y with exactly the same marginal distribution, while forcing their difference X-Y to follow a chosen centered noise or residual law. Insert the pair into a residual, VAE, or diffusion block so that the model receives the desired perturbation without changing the marginal latent distribution at either endpoint. This creates a controlled alternative to independently sampled noise, especially when marginal drift in repeated stochastic layers is harmful.

## Mathematical statement

Let μ be a probability law on R with mean zero, p_0=μ({0}), μ_+ be μ restricted to (0,∞), μ_- = (-id)#(μ restricted to (-∞,0)), and a=∫_0^∞ x μ_+(dx)=∫_0^∞ y μ_-(dy). The construction defines a joint law ν for (X,Y) by placing mass p_0 at (0,0), and otherwise mixing positive magnitudes x sampled from μ_+ and y sampled from μ_- with a circle-rotation coupling on the interval [-y,x). The rotation is R_{x,y}(u)=u+x for -y≤u<0 and R_{x,y}(u)=u-y for 0≤u<x. The resulting coordinates satisfy X and Y identically in distribution, while X-Y has law μ. For empirical discrete μ, sample (x,y) with probability proportional to (x+y)μ_+(x)μ_-(y), then sample u uniformly from [-y,x); set Y=u and X=R_{x,y}(u). The difference is D=X-Y=x when u<0 and D=-y when u≥0, so the injected residual has exactly the target centered law. The useful property is simultaneous exact marginal preservation and exact residual-law realization.

## Key formulas

- $$\nu=p_{0}\delta_{(0,0)}+\frac{1}{a}\int_{(0,\infty)^{2}}\int_{-y}^{x}\delta_{(R_{x,y}u,u)}\,du\,\mu_{+}(dx)\mu_{-}(dy),\qquad R_{x,y}(u)=\begin{cases}u+x,&-y\le u<0,\\u-y,&0\le u<x,\end{cases}$$
- $$P_{X-Y}=p_{0}\delta_{0}+\frac{1}{a}\int_{(0,\infty)^{2}}\left(y\delta_{x}+x\delta_{-y}\right)\mu_{+}(dx)\mu_{-}(dy)=\mu,\qquad X\stackrel{d}{=}Y,$$
- $$q(x,y)=\frac{(x+y)\,\mu_{+}(x)\mu_{-}(y)}{\sum_{x'>0,y'>0}(x'+y')\,\mu_{+}(x')\mu_{-}(y')},\qquad u\sim\operatorname{Unif}[-y,x),\qquad D=X-Y=\begin{cases}x,&u<0,\\-y,&u\ge 0,\end{cases}$$
- $$h_{\mathrm{new}}=h+\alpha(X-Y),\qquad \mathcal{L}(X)=\mathcal{L}(Y),\qquad \mathcal{L}(\alpha(X-Y))=\mathcal{L}(\alpha D).$$

## Implementation notes

Integrate this as a stochastic residual or noise layer in a small diffusion U-Net, VAE encoder, or residual MLP. First run a short warm-up using the baseline model and collect scalar residuals r from the chosen layer. Center them as r_c=r-mean(r), estimate an empirical histogram μ, and separate positive values into μ_plus and negative magnitudes into μ_minus. Compute p0 as the zero-bin probability and estimate a=Σ_x x μ_plus(x); after centering, the negative first moment should be approximately the same. For each nonzero sample, enumerate positive and negative histogram bins and construct q(x,y) proportional to (x+y)μ_plus(x)μ_minus(y). Sample a pair (x,y) from q, then sample u uniformly on [-y,x). Set Y=u; if u<0 set X=u+x, otherwise set X=u-y. Inject D=X-Y through h_new=h+alpha*D. If paired states are useful, instead create h_X=h+alpha*X and h_Y=h+alpha*Y and share downstream weights. The code should log the empirical histogram distance between X and Y, the histogram distance between D and μ, and gradient variance. Estimate μ and q from data; do not assume the mathematical law is known analytically. Begin with a four-layer residual MLP on MNIST or CIFAR-10 and compare Gaussian noise, independently sampled matched-moment noise, and this coupling at equal parameter count and perturbation variance. Measure validation loss, reconstruction or likelihood quality, gradient variance, and marginal distribution drift across depth. Success means lower endpoint marginal drift or gradient variance and improved validation performance at equal noise scale; failure is exact residual matching without any optimization or generalization gain.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.