# Separable Ky-Fan spectral regularization

- ID: 561
- Canonical URL: https://synthcore.org/idea/561/separable-ky-fan-spectral-regularization
- API JSON: https://synthcore.org/api/idea/561.json
- API Markdown: https://synthcore.org/api/idea/561.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.07913](https://arxiv.org/abs/2607.07913)
- Category: regularization
- Solves: stability, memory, scalability
- ML areas: regularization, mlp, optimizer, attention
- Math tags: spectral-theory, linear-algebra, tensor-decomposition
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Represent a large positive semidefinite neural operator as the sum of two Kronecker products and regularize an efficiently computed upper bound on its largest eigenvalues. The bound controls not only the spectral norm but every top-k eigenvalue sum, allowing a tunable penalty on concentrated or unstable directions without constructing the exponentially larger operator.

## Mathematical statement

Theorem 1.1 states that for positive semidefinite operators A_i and B_i acting on C^{d_i}, the eigenvalue vector of M = tensor product over i of A_i plus tensor product over i of B_i satisfies lambda(M) preceq tensor product over i of lambda(A_i) plus tensor product over i of lambda(B_i). Here lambda(X) is the vector of eigenvalues of X sorted in nonincreasing order, tensor products of vectors contain all products of one eigenvalue from each factor, and the vector sum is componentwise. For equal-length vectors x and y, x preceq y means that the sum of the k largest entries of x is at most the sum of the k largest entries of y for every k, with equality for the full dimension. Consequently, every Ky Fan spectral sum obeys ||M||_(k) = sum_{j=1}^k lambda_j(M) <= sum_{j=1}^k c_j^{downarrow}, where c = tensor product_i lambda(A_i) + tensor product_i lambda(B_i). The k=1 case gives lambda_max(M) <= product_i lambda_max(A_i) + product_i lambda_max(B_i). The useful property is that these bounds require only spectra of small PSD factors, not eigendecomposition or materialization of the large operator.

## Key formulas

- $$\lambda\left(\bigotimes_{i=1}^{n}A_i+\bigotimes_{i=1}^{n}B_i\right)\preceq\bigotimes_{i=1}^{n}\lambda(A_i)+\bigotimes_{i=1}^{n}\lambda(B_i).$$
- $$\sum_{j=1}^{k}\lambda_j\left(\bigotimes_{i=1}^{n}A_i+\bigotimes_{i=1}^{n}B_i\right)\leq\sum_{j=1}^{k}c_j^{\downarrow},\qquad c=\bigotimes_{i=1}^{n}\lambda(A_i)+\bigotimes_{i=1}^{n}\lambda(B_i).$$
- $$\lambda_{\max}(M)\leq\prod_{i=1}^{n}\lambda_{\max}(A_i)+\prod_{i=1}^{n}\lambda_{\max}(B_i),\qquad M=\bigotimes_{i=1}^{n}A_i+\bigotimes_{i=1}^{n}B_i.$$
- $$\mathcal{R}_k=\frac{1}{k}\sum_{j=1}^{k}c_j^{\downarrow},\qquad \mathcal{R}_{\mathrm{tail}}=\sum_{j>k}c_j^{\downarrow}.$$

## Implementation notes

1. Integration point: apply the method to a tensorized linear or MLP layer, a Kronecker-factored optimizer state, or a PSD attention kernel. Parameterize the large PSD operator as M = kron(A_1,...,A_n) + kron(B_1,...,B_n), with A_i = L_{A_i}L_{A_i}^T + epsilon I and B_i = L_{B_i}L_{B_i}^T + epsilon I to guarantee positive semidefiniteness. Keep the factors as the trainable representation and never materialize M during normal training. 2. Pseudocode: at each regularization interval, compute a_i = eigvalsh(A_i) and b_i = eigvalsh(B_i), sort each descending, form c = kron(a_1,...,a_n) + kron(b_1,...,b_n), sort c descending, and calculate R_k = mean(c[:k]); add lambda_reg * R_k to the task loss. For a lower-cost controller use R_1 = product_i a_i[0] + product_i b_i[0], then multiply the layer output or its learning rate by min(1, target/R_1). 3. The theorem supplies the upper bound ||M||_(k) <= sum_{j<=k} c_j, where ||M||_(k) is the sum of the k largest eigenvalues. Compute factor spectra exactly for small factors; for larger factors use Lanczos to obtain the top r eigenvalues and approximate the Kronecker spectrum from those values. Periodically materialize M in a small diagnostic model to measure bound tightness. 4. First experiment: train a tensorized MNIST or CIFAR-10 MLP with a 256x256 PSD feature metric represented by n=4 factors of size 4x4 and two Kronecker terms. Compare no spectral control, Frobenius regularization, and the proposed R_1 or R_k penalty at equal parameter count. Record loss curves, gradient-norm spikes, validation accuracy, wall-clock overhead, and the ratio of the exact operator norm to R_1. Success means fewer exploding updates or better validation accuracy at comparable compute, with factor-only spectral control much cheaper than decomposing the full matrix.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built separable PSD Kronecker-factor parameterization with exact factor-spectrum Ky-Fan/R1 bounds, plus a reproducible fitting experiment. The majorization claim held in all 100 trials with zero failures and only 3.2e-12 worst numerical slack; the R1 penalty lowered peak exact norm from 2.68 to 2.17, but final loss worsened from 0.0621 to 0.0748 and runtime rose from 11.0s to 14.6s. Thus the claimed spectral-control effect was observed, but this small task did not show a validation or optimization win.

### Mechanism check

- Verdict: Built separable PSD Kronecker-factor parameterization with exact factor-spectrum Ky-Fan/R1 bounds, plus a reproducible fitting experiment. The majorization claim held in all 100 trials with zero failures and only 3.2e-12 worst numerical slack; the R1 penalty lowered peak exact norm from 2.68 to 2.17, but final loss worsened from 0.0621 to 0.0748 and runtime rose from 11.0s to 14.6s. Thus the claimed spectral-control effect was observed, but this small task did not show a validation or optimization win.
- Confidence: 8/10
- Limitations: Only a tiny synthetic 27-dimensional tensorized regression task was tested; no MNIST/CIFAR, large operator, top-k penalty comparison, wall-clock factorization scaling study, or exploding-update regime was evaluated. The controller variant was exploratory and is not used as evidence for the penalty result.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/1388/bench_report.json)
- [bench_run.py](https://synthcore.org/code/1388/bench_run.py)
- [experiment.py](https://synthcore.org/code/1388/experiment.py)
- [report.md](https://synthcore.org/code/1388/report.md)
- [report_bench_2026-09-04T182014.md](https://synthcore.org/code/1388/report_bench_2026-09-04T182014.md)
- [results.json](https://synthcore.org/code/1388/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1388)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.