Gaussian Moment-Window Activation Regularizer

Integrate the regularizer after LayerNorm or RMSNorm and before the next linear or attention projection. For a minibatch activation tensor h with shape [B,T,d], flatten non-padding tokens into vectors h_i, compute a minibatch mean mu and per-coordinate variance v, and whiten using \widetilde h_i=(h_i-mu)/(sqrt(v)+epsilon); use stop-gradient on mu and v initially so the penalty cannot trivially reduce itself by changing the whitening statistics. Choose Q={2,4,8}, but cap it at q_max=max(2,floor(c0 sqrt(log(e d)))) with c0 in [0.5,2]. Pseudocode: compute r_i=norm(whiten(h_i),2); for each q in Q compute m_q=(mean(r_i^q))^(1/q); compute g_q(d) from the gamma formula or a cached Monte Carlo table; set lower_q=a*g_q(d)/sqrt(log(e d)) and upper_q=b*sqrt(log(e d))*g_q(d); add lambda times the sum of softplus(lower_q-m_q)+softplus(m_q-upper_q) to the task loss. The theorem supplies the comparison structure and moment window; a, b, lambda, epsilon, and q_max are empirical hyperparameters because the universal constants and log-concavity are not known for learned features. Start with a 6-layer transformer or 4-layer MLP on CIFAR-10 or WikiText-2, comparing no activation regularizer and standard LayerNorm at equal training FLOPs. Measure gradient-norm variance, the fraction of batches violating each bound, loss spikes, validation loss, and robustness to a 2-4x learning-rate increase. A useful signal is fewer activation and gradient spikes with equal-or-better validation loss at the same compute; reject the method if it merely rescales activations or adds instability.