✗ Failed on benchmark 2026

Hard-edge inverse-spectrum regularizer

Implementation & benchmark of arXiv:2607.09547 — High-Dimensional Interpolators Can Be Fragile: Heavy Tails and High-Dimensional Large Deviations

Usefulness7/10

Difficulty5/10

Novelty6/10

Regularization Stability Generalization Accuracy Regularization Loss Training dynamics Random matrix Linear algebra Statistics Probability

Source paper: High-Dimensional Interpolators Can Be Fragile: Heavy Tails and High-Dimensional Large Deviations arXiv:2607.09547 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a tail-risk penalty whenever a neural network's learned feature covariance has excessive inverse-eigenvalue mass. The penalty suppresses nearly singular representation directions, which may be inconspicuous in mean validation loss but can produce rare, very large prediction errors under noise or distribution shift.

Formulas

$$R_n=b_n+H_n,\qquad H_n=\frac{1}{n}\sum_i\lambda_{i,n}^{-1}.$$

$$\eta_n:=n^{-\chi},\quad 0<\chi<\frac14,\qquad M_n:=\frac{1}{n}\sum_{\lambda_{i,n}\geq\eta_n}\frac{1}{\lambda_{i,n}},\qquad S_n:=\frac{1}{n}\sum_{\lambda_{i,n}<\eta_n}\frac{1}{\lambda_{i,n}},\quad H_n=M_n+S_n.$$

$$\{H_n>h_{\kappa,\rho}+\delta\}\subseteq\{M_n>h_{\kappa,\rho}+\delta/2\}\cup\{S_n>\delta/2\}.$$

$$\mathcal{L}_{\mathrm{tail}}=\mathcal{L}_{\mathrm{task}}+\alpha\,\hat H+\beta\,\hat S,\qquad \hat H=\frac{1}{d}\sum_{j=1}^{d}\frac{1}{\hat\lambda_j},\quad \hat S=\frac{1}{d}\sum_{j=1}^{d}\frac{g(\hat\lambda_j;\eta)}{\hat\lambda_j},\quad \hat\lambda_j=\lambda_j\left(\frac{1}{B}Z^\top Z+\varepsilon I_d\right).$$

Mathematical statement

The paper decomposes ridgeless population risk as $R_n=b_n+H_n$, where $b_n$ is a deterministic bias term and $H_n=(1/n)\sum_i\lambda_{i,n}^{-1}$ is the positive-spectrum inverse trace of a sample covariance matrix with eigenvalues $\lambda_{i,n}>0$. For $\eta_n=n^{-\chi}$, $0<\chi<1/4$, it splits $H_n=M_n+S_n$, where $M_n=(1/n)\sum_{\lambda_{i,n}\geq\eta_n}\lambda_{i,n}^{-1}$ is the nonmicroscopic contribution and $S_n=(1/n)\sum_{\lambda_{i,n}<\eta_n}\lambda_{i,n}^{-1}$ is the microscopic hard-edge contribution. The event inclusion $\{H_n>h_{\kappa,\rho}+\delta\}\subseteq\{M_n>h_{\kappa,\rho}+\delta/2\}\cup\{S_n>\delta/2\}$ shows that an excessive inverse trace is attributable either to broad spectral drift or to very small eigenvalues. For a feature matrix $Z\in\mathbb{R}^{B\times d}$, use $C=Z^\top Z/B+\varepsilon I_d$, eigenvalues $\hat\lambda_j$, and the empirical inverse trace $\hat H=d^{-1}\sum_j\hat\lambda_j^{-1}$ plus hard-edge mass $\hat S=d^{-1}\sum_j\mathbf{1}\{\hat\lambda_j<\eta\}\hat\lambda_j^{-1}$.

Implementation notes

Integrate the regularizer at the representation immediately before a linear prediction head. For a minibatch, obtain features Z = backbone(x) with shape [B,d], subtract the batch mean from each feature dimension, and form C = Z.T @ Z / B + eps * I. For small or medium d, compute eigenvalues with a symmetric eigendecomposition; for large d, approximate the inverse trace using Hutchinson probes: H ≈ mean_v v.T @ solve(C,v) for Rademacher vectors v. Set eta = c * median(lam) with c initially in [1e-3,1e-1]. Compute H = mean(1/(lam+eps)). Replace the discontinuous hard-edge indicator with g(lam;eta)=sigmoid((eta-lam)/tau) and compute S=mean(g(lam;eta)/(lam+eps)). Optimize loss = task_loss + alpha*H + beta*S; optionally stop gradients through eta and the batch median. Pseudocode: Z=features(x); C=Z.T@Z/B+eps*I; lam=eigh(C); eta=c*median(lam); H=mean(1/(lam+eps)); S=mean(sigmoid((eta-lam)/tau)/(lam+eps)); loss=task+alpha*H+beta*S; loss.backward(); optimizer.step(). The paper supplies the inverse-spectrum decomposition and hard-edge rationale; alpha, beta, c, tau, and eps are empirical. First test a width-512 MLP or small ResNet on CIFAR-10 against cross-entropy, weight decay, and orthogonal-feature regularization. Evaluate clean accuracy, per-example-loss 99th and 99.9th percentiles, maximum logit norm, smallest covariance eigenvalue, and corruption accuracy. Success is lower extreme-loss quantiles and fewer exploding logits at comparable clean accuracy, with tolerable eigendecomposition overhead.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built the hard-edge inverse-spectrum regularizer with centered feature covariance, eigendecomposition, detached median-based threshold, smooth hard-edge gate, and a matched toy classifier comparison. The mechanism checks confirmed exact H=M+S decomposition and event inclusion, inverse hard-edge mass scaling with slope -1.000 versus -1 predicted, smooth-gate midpoint 0.5, and inverse-gradient scaling -1.966 versus -2 predicted. In the toy model, the regularizer substantially increased the minimum feature covariance eigenvalue and improved shifted accuracy and 99.9th-percentile loss, but slightly reduced clean accuracy and worsened the 99th-percentile loss, so the practical win is mixed rather than universal.

Agent confidence: 8/10

Baseline: clean accuracy 0.8475; shifted accuracy 0.8400; shifted loss q99 2.3113; q999 3.4509; max clean logit norm 3.8214; minimum feature covariance eigenvalue 0.000409
Idea: clean accuracy 0.8400; shifted accuracy 0.8450; shifted loss q99 2.3957; q999 3.3653; max clean logit norm 3.9665; minimum feature covariance eigenvalue 0.006659
Mechanism predictions: H=M+S identity error 0; event-inclusion failures 0; S slope -1.000 observed vs -1 predicted; inverse-gradient slope -1.966 observed vs -2 predicted; sigmoid gate at eta 0.5

Limitations: Only a small synthetic classification task and one fixed seed were tested; CIFAR-10, multiple seeds, FLOP/overhead measurements, large-width Hutchinson approximation, and broad hyperparameter tuning were not tested.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.09547) Sep 2, 2026

Ahead of the baseline (−0.0%) but not statistically significant (p=0.4228, wins 5 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 14.4213
Idea mean: 14.4192
Effect (Δ): -0.002 (−0.0%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.4228 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±0.0%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched tabular track was tested; vision, sequence, and dynamics tracks were not tested. The benchmark used the exact small mlp_tiny architecture, 18 epochs, 400 training and 400 test samples, and a three-value learning-rate sweep; broader alpha/beta/c/temperature sweeps, Hutchinson approximation, corruption or distribution-shift evaluation, and wall-clock overhead were not measured.

How to run: python3 bench_experiment.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Built the hard-edge inverse-spectrum regularizer into a matched mlp_tiny training system on the structurally appropriate tabular regularization track. Across 8 paired seeds, the idea improved test MSE by only 0.00204 (14.41924 vs 14.42128), with permutation p=0.4228, so there is no significant win. The trained-model mechanism signature was not confirmed: inverse trace decreased negligibly while the minimum eigenvalue was unchanged or marginally lower.

Baseline: Friedman#1 tabular mlp_tiny, lr sweep {0.001, 0.002, 0.003}, best lr=0.003; 8-seed test MSE mean 14.421282, std 1.372874; per-seed [13.569253, 14.291465, 14.336907, 13.301629, 12.592417, 14.114306, 16.196705, 16.967575]
Baseline sweep: lr=0.001 mean 25.191289; lr=0.002 mean 16.454425; lr=0.003 mean 13.874814 on sweep seeds 0-3
Idea: Matched mlp_tiny with centered covariance inverse-spectrum penalty, same lr sweep and best lr=0.003; 8-seed test MSE mean 14.419240, std 1.375494; paired delta -0.002042, 5/8 wins, permutation p=0.4228; per-seed [13.567512, 14.285934, 14.334574, 13.298772, 12.593015, 14.100104, 16.203156, 16.970856]
Idea sweep: lr=0.001 mean 25.821486; lr=0.002 mean 16.820820; lr=0.003 mean 14.419240 on 8 final seeds
Mechanism signature: Measured on trained models: baseline test inverse trace H=217.196468 and minimum eigenvalue=0.0020000001; idea H=217.114485 and minimum eigenvalue=0.0019999713; predicted H reduction occurred only at 0.038%, minimum-eigenvalue increase did not occur; confirmed=false

Benchmark result