Fourth-Mass Regularization for Signed Projections

Integrate this at a stochastic signed linear projection or quantization module, not at an arbitrary dense layer. For an input tensor \(x\in\mathbb{R}^{B\times n}\), use a coefficient vector \(a\in\mathbb{R}^n\) to form \(z_b=\sum_i a_i x_{b,i}\varepsilon_{b,i}\), with fresh independent signs \(\varepsilon_{b,i}\) per batch element or per training step. Parameterize \(a_i=u_i/\sqrt{\sum_j u_j^2+10^{-8}}\) so the required \(\ell_2\) normalization is automatic. Choose a target effective dimension \(d_0\), set \(\tau=1/d_0\), and add \(\lambda(\sum_i a_i^4-\tau)^2\) to the task loss. Pseudocode: compute a = u / sqrt(sum(u*u)+eps); compute m4 = sum(a**4); sample eps_sign = 2*Bernoulli(0.5)-1; compute z = sum(a*x*eps_sign, dim=-1); optimize task_loss + lambda*(m4-tau)**2. The paper supplies the exact fourth-moment dependence; the target \(d_0\), penalty weight \(\lambda\), and whether signs are per-example or shared are empirical choices. Estimate realized behavior by measuring batch excess kurtosis and comparing it with \(3-2m_4\) after normalizing each projection to unit variance. First test on CIFAR-10 with a small MLP or ResNet-18 containing one signed projection, comparing no penalty, an \(\ell_2\)-only baseline, and the fourth-mass penalty at matched parameter count and FLOPs. Sweep \(d_0\in\{4,8,16,n\}\). Success is lower activation-gradient variance, fewer unstable runs, and equal or better validation accuracy at the same injected-noise level; a secondary test should compare quantization error or perturbation robustness against a dense Gaussian-noise baseline.