# Spread-complexity spectral regularizer

- ID: 1943
- Canonical URL: https://synthcore.org/idea/1943/spread-complexity-spectral-regularizer
- API JSON: https://synthcore.org/api/idea/1943.json
- API Markdown: https://synthcore.org/api/idea/1943.md
- Verification status: unverified
- Source: [arXiv:2608.07412](https://arxiv.org/abs/2608.07412)
- Category: regularization
- Solves: generalization, stability
- ML areas: attention, regularization, embedding, training-dynamics
- Math tags: spectral-theory, statistics, random-matrix, harmonic-analysis, linear-algebra
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Regularize the eigenvalue spectrum of a neural representation or attention Gram matrix using the paper's universal-kernel spread-complexity curve. The loss penalizes spectral profiles that exhibit excessive level clustering or near-degeneracy, while allowing the desired amount of eigenvalue repulsion to be selected by a GOE-like, Poisson-like, or empirically calibrated target.

## Mathematical statement

The paper's central approximation is $\langle C_{\rm unfold}(t)\rangle/D\simeq \frac{1}{2}\left(1-2\sum_{k=1}^{\infty}\frac{\Phi_k(t)}{4k^2-1}\right)$, where $D$ is the matrix or Hilbert-space dimension, $C_{\rm unfold}(t)$ is spread complexity after unfolding the spectrum, and $\Phi_k(t)$ is the Fourier transform of the probability distribution of the unfolded $k$-th-nearest-neighbor spacing. For a finite neural tensor, let $\lambda_1\leq\cdots\leq\lambda_m$ be eigenvalues of a positive semidefinite feature Gram matrix $G$, let $\bar\delta$ be the robust local mean eigenvalue spacing, and define unfolded levels $x_i=\lambda_i/\bar\delta$. The empirical $k$-spacing characteristic function is $\widehat\Phi_k(t)=\frac{1}{m-k}\sum_{i=1}^{m-k}\cos\!\left(t(x_{i+k}-x_i)\right)$. Truncating at $K$ gives $\widehat c(t)=\frac12\left(1-2\sum_{k=1}^{K}\widehat\Phi_k(t)/(4k^2-1)\right)$. The regularizer compares $\widehat c(t)$ with a target curve $c_\star(t)$ using $L_{\rm spread}=\frac{1}{|T|}\sum_{t\in T}w_t[\widehat c(t)-c_\star(t)]^2$. The universal-kernel property supplies the fixed coefficients, while the spacing Fourier transforms carry the representation's local spectral statistics.

## Key formulas

- $$\frac{\langle C_{\rm unfold}(t)\rangle}{D}\simeq\frac{1}{2}\left(1-2\sum_{k=1}^{\infty}\frac{\Phi_{k}(t)}{4k^{2}-1}\right).$$
- $$\widehat\Phi_k(t)=\frac{1}{m-k}\sum_{i=1}^{m-k}\cos\!\left(t\,[x_{i+k}-x_i]\right),\qquad x_i=\frac{\lambda_i}{\bar\delta},$$
- $$\widehat c_K(t)=\frac12\left(1-2\sum_{k=1}^{K}\frac{\widehat\Phi_k(t)}{4k^2-1}\right),\qquad L_{\rm spread}=\frac{1}{|T|}\sum_{t\in T}w_t\left[\widehat c_K(t)-c_\star(t)\right]^2.$$

## Implementation notes

(1) Integration point: insert the regularizer after the projection producing a batch representation $Z\in\mathbb{R}^{B\times d}$, or after an attention head's token-score matrix. Form a centered feature Gram matrix $G=(Z-\mathbf{1}\mu^\top)^\top(Z-\mathbf{1}\mu^\top)/(B-1)+\epsilon I_d$ and use its $m=d$ eigenvalues. Compute this term every 50-200 optimizer steps to limit overhead. (2) Pseudocode: `Zc = Z - mean(Z, dim=0); G = Zc.T @ Zc/(B-1) + eps*I; lam = sort(eigh(G).values); delta = median(lam[1:] - lam[:-1]) + eps; x = lam/delta; for t in T: for k in 1..K: phi[k,t] = mean(cos(t*(x[k:] - x[:-k]))); c[t] = 0.5*(1 - 2*sum(phi[k,t]/(4*k*k-1) for k in 1..K)); Lspread = mean(w*(c-target)**2); loss = task_loss + alpha*Lspread`. Use $K=3$ and 8-16 values of $t$ initially. (3) The coefficients $1/(4k^2-1)$ and the complexity expression come from the paper; eigenvalues, robust spacing scale $\bar\delta$, target curve, and weight $\alpha$ are engineering choices. Normalize with a central quantile range if outliers dominate. Retain gradients through the eigendecomposition initially; if gradients through sorting are unstable, use a detached spectrum as a diagnostic and replace hard ordering with a differentiable soft-ranking operator. Set $c_\star$ using a simulated GOE spectrum, an isotropic Gaussian-feature baseline, or the running spectrum of a well-performing reference model. (4) First experiment: train a 6-layer, 8-head Transformer on CIFAR-10 or a small language-model corpus, comparing task loss plus this term with no spectral regularizer and with VICReg-style covariance regularization. Track validation accuracy, effective rank, minimum adjacent spacing, attention entropy, and gradient-norm variance. Success means reduced rank collapse and smoother training at equal accuracy, or better validation accuracy at equal parameter count; reject it if it only adds eigendecomposition cost without changing spectral collapse or generalization.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.