Spread-complexity spectral regularizer

(1) Integration point: insert the regularizer after the projection producing a batch representation $Z\in\mathbb{R}^{B\times d}$, or after an attention head's token-score matrix. Form a centered feature Gram matrix $G=(Z-\mathbf{1}\mu^\top)^\top(Z-\mathbf{1}\mu^\top)/(B-1)+\epsilon I_d$ and use its $m=d$ eigenvalues. Compute this term every 50-200 optimizer steps to limit overhead. (2) Pseudocode: Zc = Z - mean(Z, dim=0); G = Zc.T @ Zc/(B-1) + eps*I; lam = sort(eigh(G).values); delta = median(lam[1:] - lam[:-1]) + eps; x = lam/delta; for t in T: for k in 1..K: phi[k,t] = mean(cos(t*(x[k:] - x[:-k]))); c[t] = 0.5*(1 - 2*sum(phi[k,t]/(4*k*k-1) for k in 1..K)); Lspread = mean(w*(c-target)**2); loss = task_loss + alpha*Lspread. Use $K=3$ and 8-16 values of $t$ initially. (3) The coefficients $1/(4k^2-1)$ and the complexity expression come from the paper; eigenvalues, robust spacing scale $\bar\delta$, target curve, and weight $\alpha$ are engineering choices. Normalize with a central quantile range if outliers dominate. Retain gradients through the eigendecomposition initially; if gradients through sorting are unstable, use a detached spectrum as a diagnostic and replace hard ordering with a differentiable soft-ranking operator. Set $c_\star$ using a simulated GOE spectrum, an isotropic Gaussian-feature baseline, or the running spectrum of a well-performing reference model. (4) First experiment: train a 6-layer, 8-head Transformer on CIFAR-10 or a small language-model corpus, comparing task loss plus this term with no spectral regularizer and with VICReg-style covariance regularization. Track validation accuracy, effective rank, minimum adjacent spacing, attention entropy, and gradient-norm variance. Success means reduced rank collapse and smoother training at equal accuracy, or better validation accuracy at equal parameter count; reject it if it only adds eigendecomposition cost without changing spectral collapse or generalization.