A Correlation-Gap Bound for Nonlinear Gaussian PCA
arXiv:2607.15035
2026
Architecture
1 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper gives a mathematically sharp justification for using a fixed Karhunen–Loève/PCA basis before applying per-sample top-k sparsification. For centered Gaussian data with ordered coordinate variances, no other orthogonal rotation can achieve lower expected residual energy when the retained coordinate subset is selected adaptively after observing each sample. This can be transferred directly into sparse bottlenecks, activation compression, and token-wise MLP feature selection by using a covariance-eigenbasis as the analysis transform rather than learning an unrestricted rotation. The most useful experiment is to compare PCA-top-k against learned orthogonal transforms and random rotations at equal active-feature count.
Ideas from this paper
✗ Failed on benchmark
2026
Construct a neural activation bottleneck by projecting hidden states into a fixed covariance-eigenbasis and retaining only the d largest-magnitude coordinates per sample. For Gaussian, decorrelated activations, the paper proves that adaptive top-d selection in the PCA basis has no greater expected residual energy than adaptive top-d selection after any other orthogonal rotation. This provides a principled alternative to learning an unrestricted rotation before sparsification.
Useful7/10
Difficulty3/10
Novelty5/10