# Conserved Poisson Feature Noise

- ID: 530
- Canonical URL: https://synthcore.org/idea/530/conserved-poisson-feature-noise
- API JSON: https://synthcore.org/api/idea/530.json
- API Markdown: https://synthcore.org/api/idea/530.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.02667](https://arxiv.org/abs/2607.02667)
- Category: regularization
- Solves: generalization, stability, sample-efficiency
- ML areas: transformer, attention, cnn, regularization
- Math tags: stochastic-processes, pde, probability
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace iid dropout or iid activation noise on spatial tokens with fluctuations generated by a conserved diffusing density. Each token receives a positive mass variable whose total mass is preserved, while Poissonian stochastic flux produces correlated perturbations that explore coherent local patterns rather than independently corrupting every feature. The density is autonomous and detached from autograd, so the regularizer adds little computational overhead.

## Mathematical statement

The paper defines a conserved density ρ(x,t) through the continuity equation ∂ₜρ = −∇·J with flux J = −D₀∇ρ + √(2D₀ρ) ζ. Here ρ(x,t)≥0 is the density, J is the mass flux, D₀>0 is the diffusion constant, and ζ is Gaussian white noise satisfying ⟨ζᵢ(x,t)ζⱼ(x′,t′)⟩ = δᵢⱼ δᵈ(x−x′)δ(t−t′). The divergence form preserves total mass under periodic or no-flux boundaries, while √ρ creates state-dependent Poisson-like fluctuations. The paper argues that coupling this density to an order parameter produces noise white in time but long-range in space. We discretize the construction on a token graph. Let ρ∈Rᴺ₊ be one density per token, L the graph Laplacian, B an oriented edge-incidence matrix satisfying BBᵀ=L, ξ standard-normal edge noise, and Π₊ a nonnegative projection that restores total mass N. The Euler-Maruyama update is ρᵗ⁺¹ = Π₊[(I−ΔtD₀L)ρᵗ + √(2D₀Δt) B diag(√ρᵗ_edge) ξᵗ]. The perturbation is hᵢ′ = hᵢ + α(ρᵢ/barρ−1)gᵢ, where hᵢ is the token feature, barρ=N⁻¹Σᵢρᵢ, gᵢ is fresh channelwise Gaussian noise, and α controls strength.

## Key formulas

- $$∂ₜρ=−∇·J,\qquad J=−D₀∇ρ+√(2D₀ρ)ζ$$
- $$⟨ζᵢ(x,t)ζⱼ(x′,t′)⟩=δᵢⱼδᵈ(x−x′)δ(t−t′)$$
- $$ρ^{t+1}=Π₊[(I−ΔtD₀L)ρ^t+√(2D₀Δt)B\,diag(√ρ^t_edge)ξ^t],\qquad BBᵀ=L$$
- $$h_i′=h_i+α(ρ_i/barρ−1)g_i,\qquad barρ=N^{-1}Σ_{i=1}^Nρ_i$$

## Implementation notes

Integrate this at the residual-stream input of a vision transformer, patch transformer, or CNN feature grid during training only. Build a fixed sparse graph over spatial tokens: use a four-neighbor grid for images, or a local-window/ring graph for sequences. Store its oriented edge-incidence matrix B and Laplacian L=B Bᵀ. Maintain one scalar density vector ρ of length N per batch example, initialized to ones with total mass N. At each training step, sample standard-normal edge noise ξ and channel noise g. Compute edge noise as √ρ_edge times ξ, scatter it through B, and update ρ using the displayed Euler-Maruyama rule. Use ΔtD₀λ_max(L)≤1 for an initial stable step-size choice. Clamp negative entries to zero and rescale the vector so its sum is exactly N. Detach ρ from autograd. Add α(ρ−1)g to patch embeddings or residual activations, optionally after normalizing g so the added variance matches the iid Gaussian baseline. Pseudocode: initialize ρ=ones; sample ξ,g; ρ_tmp=ρ−dtD₀(Lρ)+sqrt(2D₀dt)B@(sqrt(ρ_edge)*ξ); ρ=positive_mass_projection(ρ_tmp,N); h=h+α(ρ−1)[:,None]*g; run the network and backpropagate. The mathematical ingredients are the conservative diffusion and square-root noise; estimate activation variance, token covariance, and validation metrics empirically. First test ViT-Tiny on CIFAR-100 or ImageNet-100 against no noise, dropout, and iid Gaussian activation noise at matched variance and FLOPs. Success means lower validation error, better corruption robustness, or equally good accuracy with fewer training steps, together with stable loss curves and longer-range token covariance than iid noise.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a conserved graph-density noise module with Euler–Maruyama flux, positivity projection, exact mass preservation, and a small digits-token classifier comparison. The numerical check verified nonnegativity and conservation with maximum mass error 2.8e-14 and showed nonzero spatial covariance unlike iid noise. In one matched-variance run, conserved noise reached 95.56% accuracy versus 94.67% for iid noise and 95.33% for no noise, but the small 0.22 percentage-point gain over baseline was not established as reproducible and incurred about 6.5x the training time, so there is no sufficiently clear overall win.

### Mechanism check

- Verdict: Built a conserved graph-density noise module with Euler–Maruyama flux, positivity projection, exact mass preservation, and a small digits-token classifier comparison. The numerical check verified nonnegativity and conservation with maximum mass error 2.8e-14 and showed nonzero spatial covariance unlike iid noise. In one matched-variance run, conserved noise reached 95.56% accuracy versus 94.67% for iid noise and 95.33% for no noise, but the small 0.22 percentage-point gain over baseline was not established as reproducible and incurred about 6.5x the training time, so there is no sufficiently clear overall win.
- Confidence: 8/10
- Limitations: Only the sklearn 8x8 digits dataset and a small MLP token model were tested, with one random seed and 15 epochs. No ViT/CIFAR corruption robustness, multi-seed confidence intervals, FLOP-normalized comparison, or larger spatial graphs were evaluated; the density update is a dense NumPy implementation rather than a production sparse kernel.

## Artifacts

- [experiment.py](https://synthcore.org/code/1373/experiment.py)
- [report.md](https://synthcore.org/code/1373/report.md)
- [results.json](https://synthcore.org/code/1373/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1373)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.