# Jacobi Moment Spectral Regularizer

- ID: 1480
- Canonical URL: https://synthcore.org/idea/1480/jacobi-moment-spectral-regularizer
- API JSON: https://synthcore.org/api/idea/1480.json
- API Markdown: https://synthcore.org/api/idea/1480.md
- Verification status: unverified
- Source: [arXiv:2607.27185](https://arxiv.org/abs/2607.27185)
- Category: regularization
- Solves: stability, generalization
- ML areas: mlp, transformer, regularization, training-dynamics
- Math tags: spectral-theory, linear-algebra, probability, numerical-analysis
- Ratings: usefulness 5/10; difficulty 5/10; novelty 6/10

## Idea description

Regularize the Gram spectrum of selected neural layers so that its low-order moments match the spectral moments generated by a truncated q-boson Jacobi operator. Unlike a simple Frobenius or spectral-norm penalty, this controls several parts of the singular-value distribution simultaneously and can discourage harmful spectral tails without forcing all singular values to be equal.

## Mathematical statement

Let T_L be the L-by-L truncation of the paper's tridiagonal operator with (T_L)_{r,r+1}=sqrt([r+1]_q), where [r]_q=(1-q^r)/(1-q), and let e_0=(1,0,...,0)^T. Define target even moments a_{2ell}=e_0^T T_L^{2ell}e_0 and normalize them by a_2 so that the target second moment is one: mu_{2ell}=a_{2ell}/a_2^ell. For a neural weight matrix W with input width d, define the normalized Gram matrix C=W^T W/(tr(W^T W)/d). Its empirical normalized moments are b_ell=tr(C^ell)/d. Add R(W)=sum_{ell=2}^p beta_ell(b_ell-mu_{2ell})^2, optionally including a penalty on b_1-1. Trace powers can be estimated without forming C using Hutchinson probes z satisfying E[zz^T]=I_d: tr(C^ell)/d is estimated by (1/(Kd)) sum_i z_i^T C^ell z_i. The transferable mathematical asset is the explicit moment sequence induced by a Jacobi recurrence; the neural adaptation treats it as a target spectral shape.

## Key formulas

- $$T_L=\begin{pmatrix}0&\sqrt{[1]_q}&0&\cdots\\\sqrt{[1]_q}&0&\sqrt{[2]_q}&\cdots\\0&\sqrt{[2]_q}&0&\ddots\\\vdots&\vdots&\ddots&\ddots\end{pmatrix},\qquad a_{2\ell}=\langle f_0,T_L^{2\ell}f_0\rangle.$$
- $$\mu_{2\ell}=\frac{a_{2\ell}}{a_2^{\ell}},\qquad C=\frac{W^\top W}{\operatorname{tr}(W^\top W)/d},\qquad b_\ell=\frac{1}{d}\operatorname{tr}(C^\ell).$$
- $$\mathcal{R}_{\mathrm{Jacobi}}(W)=\sum_{\ell=2}^{p}\beta_\ell\bigl(b_\ell-\mu_{2\ell}\bigr)^2.$$
- $$\widehat b_\ell=\frac{1}{Kd}\sum_{i=1}^{K}z_i^\top C^\ell z_i,\qquad \mathbb{E}[z_i z_i^\top]=I_d.$$

## Implementation notes

Attach the regularizer to weight matrices of MLP expansion/projection layers or attention Q,K,V projections during training, initially excluding embeddings and output heads. Precompute target values mu_4, mu_6, and mu_8 from a 32-by-32 or 64-by-64 T_L for a selected q; these constants are detached from autograd. At each regularization step, form C implicitly. For each Hutchinson probe z, compute v_0=z and repeatedly apply v_{j+1}=W^T(Wv_j)/s, where s=tr(W^T W)/d is computed directly from the weight tensor. Estimate b_j by z^T v_j/d, average over K=1 or 2 fixed Rademacher probes, and add lambda times the squared deviations from mu_{2j}. Refresh probes every few hundred steps and backpropagate through W but not through the target moments. The paper contributes the deterministic recurrence and moment targets; trace estimation and coefficient selection are empirical. Begin with a six-layer width-256 MLP on CIFAR-10 or a two-layer transformer on a small language dataset. Compare no spectral regularization, Frobenius decay, spectral normalization, and this penalty at matched FLOPs. Track validation loss, gradient norm, estimated top singular value, and b_4,b_6,b_8. Success is reduced spectral-tail growth, improved validation performance, or stable training with a larger learning rate without divergence.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.