△ Mechanism confirmed, baseline not beaten 2026

Separable Ky-Fan spectral regularization

Implementation & benchmark of arXiv:2607.07913 — A majorization relation for a sum of two tensor products of positive semidefinite operators

Usefulness6/10

Difficulty5/10

Novelty7/10

Regularization Stability Memory Scalability Regularization Mlp Optimizer Attention Spectral theory Linear algebra Tensor decomposition

Source paper: A majorization relation for a sum of two tensor products of positive semidefinite operators arXiv:2607.07913 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Represent a large positive semidefinite neural operator as the sum of two Kronecker products and regularize an efficiently computed upper bound on its largest eigenvalues. The bound controls not only the spectral norm but every top-k eigenvalue sum, allowing a tunable penalty on concentrated or unstable directions without constructing the exponentially larger operator.

Formulas

$$\lambda\left(\bigotimes_{i=1}^{n}A_i+\bigotimes_{i=1}^{n}B_i\right)\preceq\bigotimes_{i=1}^{n}\lambda(A_i)+\bigotimes_{i=1}^{n}\lambda(B_i).$$

$$\sum_{j=1}^{k}\lambda_j\left(\bigotimes_{i=1}^{n}A_i+\bigotimes_{i=1}^{n}B_i\right)\leq\sum_{j=1}^{k}c_j^{\downarrow},\qquad c=\bigotimes_{i=1}^{n}\lambda(A_i)+\bigotimes_{i=1}^{n}\lambda(B_i).$$

$$\lambda_{\max}(M)\leq\prod_{i=1}^{n}\lambda_{\max}(A_i)+\prod_{i=1}^{n}\lambda_{\max}(B_i),\qquad M=\bigotimes_{i=1}^{n}A_i+\bigotimes_{i=1}^{n}B_i.$$

$$\mathcal{R}_k=\frac{1}{k}\sum_{j=1}^{k}c_j^{\downarrow},\qquad \mathcal{R}_{\mathrm{tail}}=\sum_{j>k}c_j^{\downarrow}.$$

Mathematical statement

Theorem 1.1 states that for positive semidefinite operators A_i and B_i acting on C^{d_i}, the eigenvalue vector of M = tensor product over i of A_i plus tensor product over i of B_i satisfies lambda(M) preceq tensor product over i of lambda(A_i) plus tensor product over i of lambda(B_i). Here lambda(X) is the vector of eigenvalues of X sorted in nonincreasing order, tensor products of vectors contain all products of one eigenvalue from each factor, and the vector sum is componentwise. For equal-length vectors x and y, x preceq y means that the sum of the k largest entries of x is at most the sum of the k largest entries of y for every k, with equality for the full dimension. Consequently, every Ky Fan spectral sum obeys ||M||_(k) = sum_{j=1}^k lambda_j(M) <= sum_{j=1}^k c_j^{downarrow}, where c = tensor product_i lambda(A_i) + tensor product_i lambda(B_i). The k=1 case gives lambda_max(M) <= product_i lambda_max(A_i) + product_i lambda_max(B_i). The useful property is that these bounds require only spectra of small PSD factors, not eigendecomposition or materialization of the large operator.

Implementation notes

1. Integration point: apply the method to a tensorized linear or MLP layer, a Kronecker-factored optimizer state, or a PSD attention kernel. Parameterize the large PSD operator as M = kron(A_1,...,A_n) + kron(B_1,...,B_n), with A_i = L_{A_i}L_{A_i}^T + epsilon I and B_i = L_{B_i}L_{B_i}^T + epsilon I to guarantee positive semidefiniteness. Keep the factors as the trainable representation and never materialize M during normal training. 2. Pseudocode: at each regularization interval, compute a_i = eigvalsh(A_i) and b_i = eigvalsh(B_i), sort each descending, form c = kron(a_1,...,a_n) + kron(b_1,...,b_n), sort c descending, and calculate R_k = mean(c[:k]); add lambda_reg * R_k to the task loss. For a lower-cost controller use R_1 = product_i a_i[0] + product_i b_i[0], then multiply the layer output or its learning rate by min(1, target/R_1). 3. The theorem supplies the upper bound ||M||_(k) <= sum_{j<=k} c_j, where ||M||_(k) is the sum of the k largest eigenvalues. Compute factor spectra exactly for small factors; for larger factors use Lanczos to obtain the top r eigenvalues and approximate the Kronecker spectrum from those values. Periodically materialize M in a small diagnostic model to measure bound tightness. 4. First experiment: train a tensorized MNIST or CIFAR-10 MLP with a 256x256 PSD feature metric represented by n=4 factors of size 4x4 and two Kronecker terms. Compare no spectral control, Frobenius regularization, and the proposed R_1 or R_k penalty at equal parameter count. Record loss curves, gradient-norm spikes, validation accuracy, wall-clock overhead, and the ratio of the exact operator norm to R_1. Success means fewer exploding updates or better validation accuracy at comparable compute, with factor-only spectral control much cheaper than decomposing the full matrix.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built separable PSD Kronecker-factor parameterization with exact factor-spectrum Ky-Fan/R1 bounds, plus a reproducible fitting experiment. The majorization claim held in all 100 trials with zero failures and only 3.2e-12 worst numerical slack; the R1 penalty lowered peak exact norm from 2.68 to 2.17, but final loss worsened from 0.0621 to 0.0748 and runtime rose from 11.0s to 14.6s. Thus the claimed spectral-control effect was observed, but this small task did not show a validation or optimization win.

Agent confidence: 8/10

Baseline: final loss 0.06210; best loss 0.05583; max exact norm 2.6799; final norm 0.7381; 250/250 steps; 11.00 s
Idea: R1 penalty: final loss 0.07479; best loss 0.06772; max exact norm 2.1670; final norm 0.7400; final exact norm/R1 0.904; 250/250 steps; 14.60 s. Verification: 0/100 failures, worst cumulative gap 3.18e-12.

Limitations: Only a tiny synthetic 27-dimensional tensorized regression task was tested; no MNIST/CIFAR, large operator, top-k penalty comparison, wall-clock factorization scaling study, or exploding-update regime was evaluated. The controller variant was exploratory and is not used as evidence for the penalty result.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.07913) Sep 4, 2026

Significantly WORSE than the tuned baseline (+22.8%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 12.8428
Idea mean: 15.7669
Effect (Δ): 2.9241 (+22.8%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±12.4%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Only the registered tabular regularization track was tested, using a small 64-dimensional hidden PSD operator, 400 training examples, and 15 epochs. The larger proposed MNIST/CIFAR setting, wall-clock overhead, exploding-update frequency, and broader factor-size scaling were not tested.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented separable Ky-Fan spectral regularization on the registered tabular/mlp_tiny benchmark. The majorization check passed all 40 trials with zero failures and worst slack 2.73e-12; the trained-model R1 signature was confirmed with mean exact-norm/R1 ratio 0.540. However, the idea was significantly worse than the tuned baseline across 8 paired seeds, so it does not constitute a benchmark win.

Baseline: test MSE mean 12.842816, std 1.379313, best cfg lr=0.006 wd=0.0001
Idea: test MSE mean 15.766903, std 1.232610, best cfg lr=0.006 wd=0.0001 lambda=0.001 k=4; paired delta +2.924086; permutation p=0.0081

Benchmark result