High-Dimensional Interpolators Can Be Fragile: Heavy Tails and High-Dimensional Large Deviations
arXiv:2607.09547
2026
Regularization
2 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper identifies a concrete failure mode of interpolation: average predictive risk can remain controlled while rare risk explosions are caused by inverse eigenvalues near the covariance hard edge. Its exact decomposition, $R_n=b_n+H_n$ with $H_n=n^{-1}\sum_i\lambda_{i,n}^{-1}$, gives a computable fragility diagnostic for neural-network features and last-layer representations. The most transferable interventions are an inverse-spectrum regularizer and an adaptive ridge controller that increases regularization when the representation Gram matrix becomes nearly singular. These methods target tail risk and stability rather than only mean validation loss.
Ideas from this paper
✗ Failed on benchmark
2026
Add a tail-risk penalty whenever a neural network's learned feature covariance has excessive inverse-eigenvalue mass. The penalty suppresses nearly singular representation directions, which may be inconspicuous in mean validation loss but can produce rare, very large prediction errors under noise or distribution shift.
Useful7/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace a fixed ridge coefficient in a neural network's final head with a controller driven by inverse spectral mass and hard-edge mass. The head can remain weakly regularized when the feature spectrum is healthy, but automatically increases ridge strength when small eigenvalues signal a high-risk interpolation regime.
Useful6/10
Difficulty4/10
Novelty5/10