# Spectral-gap-aware Jacobi whitening

- ID: 1307
- Canonical URL: https://synthcore.org/idea/1307/spectral-gap-aware-jacobi-whitening
- API JSON: https://synthcore.org/api/idea/1307.json
- API Markdown: https://synthcore.org/api/idea/1307.md
- Verification status: unverified
- Source: [arXiv:2607.23187](https://arxiv.org/abs/2607.23187)
- Category: optimization
- Solves: speedup, stability, accuracy
- ML areas: optimizer, training-dynamics, normalization, inference-speedup
- Math tags: spectral-theory, linear-algebra, numerical-analysis, optimization
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace magnitude-only pivot selection in an approximate symmetric eigensolver with a perturbation score that divides squared off-diagonal coupling by the spectral gap between the associated diagonal entries. In covariance whitening or second-order preconditioning, this should spend a limited number of rotations resolving nearly degenerate eigenspaces while ignoring harmless couplings between well-separated modes.

## Mathematical statement

The paper's motivating matrix is A = blkdiag({[[1, 10^-8], [10^-8, 2]], [[1, 10^-9], [10^-9, 1+10^-10]]}), showing that the larger off-diagonal entry can be less important because its diagonal spectral gap is large. For a symmetric matrix A = D + E, with diagonal entries d_i = A_ii and off-diagonal coupling e_ij = A_ij, second-order eigenvalue perturbation from the (i,j) coupling scales as e_ij^2 divided by (d_i-d_j). Define the nonnegative pivot score s_ij = A_ij^2/(abs(A_ii-A_jj)+epsilon), where epsilon > 0 prevents division by zero. For an exact Jacobi rotation on indices i,j, define tau = (A_jj-A_ii)/(2 A_ij), t = sign(tau)/(abs(tau)+sqrt(1+tau^2)), c = 1/sqrt(1+t^2), and s = t c. The rotation J_ij(c,s) eliminates A_ij. The adaptation uses s_ij to choose rotations under a fixed compute budget, while retaining the exact Jacobi update after a pivot is selected.

## Key formulas

- $$A=\mathsf{blkdiag}\left\{\begin{bmatrix}1&10^{-8}\\10^{-8}&2\end{bmatrix},\begin{bmatrix}1&10^{-9}\\10^{-9}&1+10^{-10}\end{bmatrix}\right\}.$$
- $$\Delta\lambda_{ij}^{(2)}\approx\frac{e_{ij}^{2}}{d_i-d_j},\qquad s_{ij}=\frac{A_{ij}^{2}}{|A_{ii}-A_{jj}|+\varepsilon}.$$
- $$\tau=\frac{A_{jj}-A_{ii}}{2A_{ij}},\qquad t=\frac{\operatorname{sign}(\tau)}{|\tau|+\sqrt{1+\tau^2}},\qquad c=\frac{1}{\sqrt{1+t^2}},\qquad s=tc.$$
- $$A\leftarrow J_{ij}(c,s)^\top A J_{ij}(c,s),\qquad J_{ij}|_{\{i,j\}\times\{i,j\}}=\begin{bmatrix}c&-s\\s&c\end{bmatrix}.$$

## Implementation notes

Integrate this into the approximate symmetric eigendecomposition used by a whitening, covariance-normalization, K-FAC-style preconditioner, or spectral-regularization layer. Given a symmetric d by d covariance or curvature proxy A, maintain an accumulated orthogonal matrix Q = I and perform only K Jacobi rotations rather than iterating to full convergence. At each step, compute candidate scores s_ij = A_ij^2/(abs(A_ii-A_jj)+epsilon), optionally mask pairs already processed in the current sweep, and select the largest score. For the selected pair, compute tau, t, c, and s using the formulas above. Update the affected rows and columns of A by A <- J transpose A J, and update Q <- Q J. After K rotations, use the diagonal of A as approximate eigenvalues and Q as approximate eigenvectors. For whitening, construct W = Q diag((diag(A)+delta)^(-1/2)) Q transpose, with delta added for positive definiteness. Use epsilon equal to 10^-6 times the mean diagonal magnitude and delta equal to 10^-5. The mathematical score and rotation are fixed by the paper; estimate A empirically from minibatch activations. Compare against magnitude pivoting based on abs(A_ij), random pivoting, and a full eigendecomposition baseline. Start with a two-layer MLP on CIFAR-10, whitening 128-dimensional hidden activations, and test K in {d, 2d, 4d} using FP16 or BF16. Measure whitening error ||W A W transpose - I||_F, downstream validation accuracy, loss descent at equal rotation count, and eigensolver wall-clock time. Success means lower whitening error or faster loss reduction than magnitude pivoting at equal K, particularly for covariance matrices with clustered eigenvalues.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.