# Stealthy Low-Frequency Activation Regularizer

- ID: 1407
- Canonical URL: https://synthcore.org/idea/1407/stealthy-low-frequency-activation-regularizer
- API JSON: https://synthcore.org/api/idea/1407.json
- API Markdown: https://synthcore.org/api/idea/1407.md
- Verification status: unverified
- Source: [arXiv:2607.25616](https://arxiv.org/abs/2607.25616)
- Category: regularization
- Solves: stability, generalization
- ML areas: cnn, moe-routing, regularization
- Math tags: spectral-theory, probability, harmonic-analysis
- Ratings: usefulness 5/10; difficulty 3/10; novelty 6/10

## Idea description

Treat active spatial sites or routed tokens as an empirical point process and penalize their Fourier power in a chosen neighborhood of zero frequency. Unlike ordinary total-variation or decorrelation penalties, this specifically suppresses large-scale count fluctuations while allowing fine-scale structure to remain, potentially stabilizing sparse routing and convolutional feature maps.

## Mathematical statement

For a translation-invariant point process \(\eta\) on \(\mathbb{R}^{d}\), the paper defines the linear statistic \(N_f(\omega)=\int f\,d\omega\), where \(\omega\) is a point configuration and \(f\) is a Schwartz test function. Its Fourier transform is \(\widehat f(\xi)=\int_{\mathbb{R}^{d}} f(x)e^{-2\pi i\langle x,\xi\rangle}\,d\lambda_d(x)\), with \(\lambda_d\) Lebesgue measure. The Bartlett spectrum \(\sigma_\eta\) is the positive translation-bounded measure satisfying \(\operatorname{Var}_\eta(N_f)=\int |\widehat f(\xi)|^2\,d\sigma_\eta(\xi)\). Stealthiness means \(\sigma_\eta(B_\varepsilon(0))=0\) for some \(\varepsilon>0\), so all sufficiently low-frequency test functions have zero counting variance. For a neural feature map, let \(z_{b,x}\in[0,1]\) be the activity or routing probability at spatial site \(x\) for sample \(b\), let \(\bar z_x=B^{-1}\sum_b z_{b,x}\), and define centered Fourier power \(P(k)=B^{-1}\sum_b|\sum_x(z_{b,x}-\bar z_x)e^{-2\pi i\langle k,x\rangle}|^2\). Penalizing \(P(k)\) for \(0<\|k\|\leq k_0\) is an empirical finite-grid analogue of forcing the Bartlett spectrum to vanish near the origin.

## Key formulas

- $$N_f(\omega):=\int_{\mathbb{R}^{d}}f\,d\omega,\qquad \widehat{f}(\xi):=\int_{\mathbb{R}^{d}}f(x)e^{-2\pi i\langle x,\xi\rangle}\,d\lambda_d(x).$$
- $$\operatorname{Var}_{\eta}(N_f)=\int_{\mathbb{R}^{d}}|\widehat{f}(\xi)|^{2}\,d\sigma_{\eta}(\xi).$$
- $$\text{stealthy}\iff \sigma_{\eta}(B_{\varepsilon}(0))=0\ \text{for some }\varepsilon>0.\qquad \text{hyperuniform}\iff \operatorname{Var}_{\eta}(\omega(B_R(0)))=o(R^d).$$
- $$\mathcal{L}_{\mathrm{stealth}}=\frac{1}{|K_0|}\sum_{k\in K_0}w_kP(k),\quad P(k)=\frac{1}{B}\sum_{b=1}^{B}\left|\sum_{x\in\Lambda}(z_{b,x}-\bar z_x)e^{-2\pi i\langle k,x\rangle}\right|^2,\quad \bar z_x=\frac1B\sum_{b=1}^{B}z_{b,x}.$$

## Implementation notes

Integrate the regularizer after a spatial activation map, sparse attention mask, or MoE routing mask has been produced. Use a tensor \(z\in\mathbb{R}^{B\times H\times W}\) for CNN activations or \(z\in\mathbb{R}^{B\times T\times E}\) for token-to-expert routing probabilities; for hard masks, compute the loss on the pre-threshold probabilities so gradients exist. First choose a low-frequency set \(K_0=\{k:0<\|k\|\leq k_0\}\), excluding the DC mode because centering already removes global mean variation. Compute \(\bar z_x\) across the batch, form \(u_{b,x}=z_{b,x}-\bar z_x\), apply a real or complex FFT over spatial/token position, and set \(\mathcal L_{stealth}=|K_0|^{-1}\sum_{k\in K_0}w_k B^{-1}\sum_b|\widehat u_b(k)|^2\). Add \(\lambda\mathcal L_{stealth}\) to the task loss, with a linear warmup of \(\lambda\) over the first 5–10% of training. The paper supplies the mathematical interpretation: Fourier power estimates the Bartlett-spectrum mass, and a low-frequency penalty approximates a spectral gap; the finite-batch FFT and frequency weights are empirical engineering choices. Start with CIFAR-10 on a small ResNet-18 using 32x32 feature maps, comparing baseline, total-variation regularization, and this loss at matched FLOPs. Measure validation accuracy, loss variance across batches, spatial count variance over large windows, and low-frequency power. A successful result is lower batch-to-batch loss/gradient variance and equal-or-better accuracy at the same training budget; for MoE, additionally require unchanged expert load balance and fewer routing overflows.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.