Stealthy Low-Frequency Activation Regularizer

Integrate the regularizer after a spatial activation map, sparse attention mask, or MoE routing mask has been produced. Use a tensor \(z\in\mathbb{R}^{B\times H\times W}\) for CNN activations or \(z\in\mathbb{R}^{B\times T\times E}\) for token-to-expert routing probabilities; for hard masks, compute the loss on the pre-threshold probabilities so gradients exist. First choose a low-frequency set \(K_0=\{k:0<\|k\|\leq k_0\}\), excluding the DC mode because centering already removes global mean variation. Compute \(\bar z_x\) across the batch, form \(u_{b,x}=z_{b,x}-\bar z_x\), apply a real or complex FFT over spatial/token position, and set \(\mathcal L_{stealth}=|K_0|^{-1}\sum_{k\in K_0}w_k B^{-1}\sum_b|\widehat u_b(k)|^2\). Add \(\lambda\mathcal L_{stealth}\) to the task loss, with a linear warmup of \(\lambda\) over the first 5–10% of training. The paper supplies the mathematical interpretation: Fourier power estimates the Bartlett-spectrum mass, and a low-frequency penalty approximates a spectral gap; the finite-batch FFT and frequency weights are empirical engineering choices. Start with CIFAR-10 on a small ResNet-18 using 32x32 feature maps, comparing baseline, total-variation regularization, and this loss at matched FLOPs. Measure validation accuracy, loss variance across batches, spatial count variance over large windows, and low-frequency power. A successful result is lower batch-to-batch loss/gradient variance and equal-or-better accuracy at the same training budget; for MoE, additionally require unchanged expert load balance and fewer routing overflows.