✗ Failed on benchmark 2026

KL-TopK Activation Bottleneck

Implementation & benchmark of arXiv:2607.15035 — A Correlation-Gap Bound for Nonlinear Gaussian PCA

Usefulness7/10

Difficulty3/10

Novelty5/10

Architecture Memory Scalability Accuracy Embedding Mlp Quantization Linear algebra Probability Convex analysis Combinatorics

Source paper: A Correlation-Gap Bound for Nonlinear Gaussian PCA arXiv:2607.15035 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Construct a neural activation bottleneck by projecting hidden states into a fixed covariance-eigenbasis and retaining only the d largest-magnitude coordinates per sample. For Gaussian, decorrelated activations, the paper proves that adaptive top-d selection in the PCA basis has no greater expected residual energy than adaptive top-d selection after any other orthogonal rotation. This provides a principled alternative to learning an unrestricted rotation before sparsification.

Formulas

$$\mathbb{E}\sum_{j=d+1}^{p}X_{[j]}^{2}\leq\mathbb{E}\sum_{j=d+1}^{p}(UX)_{[j]}^{2},\qquad U\in O(p),\quad 1\leq d<p.$$

$$\operatorname{TopD}(z)_i=z_i\,\mathbf{1}\{i\in I_d(z)\},\qquad I_d(z)\in\operatorname{arg\,max}_{|I|=d}\sum_{i\in I}|z_i|^2.$$

$$\mathcal{L}_{\mathrm{top\text{-}d}}(h)=\left\|Q^\top h-\operatorname{TopD}(Q^\top h)\right\|_2^2=\sum_{j=d+1}^{p}(Q^\top h)_{[j]}^2,\qquad CQ=Q\operatorname{diag}(\lambda_1,\ldots,\lambda_p).$$

$$\text{retained-energy comparison factor}=1+O\!\left(\frac{1}{\sqrt d}\right).$$

Mathematical statement

Let X=(X_1,...,X_p) be a centered Gaussian vector with independent coordinates and variances \lambda_1\geq\cdots\geq\lambda_p. Let z_{[1]}^2\geq\cdots\geq z_{[p]}^2 denote the decreasing rearrangement of the squared coordinates of z. For every orthogonal matrix U\in O(p) and every 1\leq d<p, the paper proves the correlation-gap inequality \mathbb{E}[\sum_{j=d+1}^{p}X_{[j]}^2]\leq\mathbb{E}[\sum_{j=d+1}^{p}(UX)_{[j]}^2]. The left-hand side is the expected squared residual after keeping the d largest coordinates of X; the right-hand side is the residual after first rotating X with U and then keeping the d largest rotated coordinates. Thus the KL basis is optimal for this adaptive nonlinear coordinate-retention objective in the stated Gaussian model. For neural activations h with covariance C, compute an eigendecomposition C=Q\operatorname{diag}(\lambda_1,...,\lambda_p)Q^ op with eigenvalues in descending order and use z=Q^ op h. The neural residual is \|z-\operatorname{TopD}(z)\|_2^2=\sum_{j=d+1}^{p}z_{[j]}^2. The paper also identifies the remaining comparison loss through the correlation gap of the rank-d uniform matroid and obtains a 1+O(1/\sqrt d) retained-energy comparison, meaning that any possible advantage of optimizing the orthogonal basis diminishes as d grows.

Implementation notes

(1) Integration point: insert the module before an activation-compression interface, sparse autoencoder decoder, MLP bottleneck, or intermediate activation stored for gradient checkpointing. Let H\in\mathbb{R}^{B imes p} be a batch of hidden states. Maintain a fixed orthogonal matrix Q and transform each row with Z=HQ. (2) Pseudocode: collect N calibration activations h_n; compute the mean \bar h and covariance C=(1/N)\sum_{n=1}^N(h_n-\bar h)(h_n-\bar h)^ op; compute Q and eigenvalues with an eigensolver; sort eigenvectors so \lambda_1\geq\cdots\geq\lambda_p. For each training or inference batch, compute Z=(H-\bar h)Q; for every row b, set I_b to the d indices with largest |Z_{b,i}|; form S_{b,i}=Z_{b,i} when i\in I_b and S_{b,i}=0 otherwise; reconstruct \widehat H=S Q^ op+\bar h if the downstream layer expects dense activations. Add \alpha\|H-\widehat H\|_2^2 to the task objective when the bottleneck is trainable. (3) Computed from the paper's mathematics: the orthogonal PCA transform, adaptive top-d rule, and exact residual-energy formula. Estimated empirically: covariance quality, deviation from Gaussianity, covariance drift during training, and the benefit of refreshing Q. In the first implementation, stop gradients through the top-d indices and keep Q fixed after calibration; a straight-through estimator can be tested later. (4) First cheap experiment: train a small Transformer on WikiText-2 or a two-layer MLP on CIFAR-10, then compare uncompressed activations, PCA-top-d, random-rotation-top-d, and a jointly learned orthogonal rotation at d/p values 0.1, 0.25, and 0.5. Measure reconstruction MSE, validation perplexity or accuracy, peak activation memory, and wall-clock time. The falsifiable target is that PCA-top-d matches or beats the learned-rotation baseline at equal d, while reducing stored activation values roughly in proportion to d/p; additionally, the performance gap between PCA and learned rotations should shrink as d increases.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a reproducible Gaussian KL-top-k activation bottleneck with calibration-time PCA, adaptive per-sample top-d retention, reconstruction, and numerical checks. The mechanism manifested: residual energy scaled linearly with covariance scale, isotropic inputs were rotation-invariant within Monte Carlo uncertainty, and PCA had lower residual than every tested random rotation across d=1..7. In the p=32 bottleneck, PCA retained 63.3%, 88.2%, and 97.8% of energy at d/p=0.094, 0.25, and 0.5, versus 39.8%, 70.6%, and 92.0% for a random rotation; the exact residual implementation error was 3.6e-15.

Agent confidence: 8/10

Baseline: Random orthogonal rotation top-d: MSE 48.10, 23.53, and 6.42 at d/p=0.094, 0.25, and 0.5; retained-energy fractions 0.398, 0.706, and 0.920.
Idea: Calibration PCA top-d: MSE 29.38, 9.42, and 1.78 at d/p=0.094, 0.25, and 0.5; retained-energy fractions 0.633, 0.882, and 0.978. Scale sweep residual/unit-scale ratios were 0.25, 0.5, 1, 2, and 4 as predicted.

Limitations: This is a synthetic Gaussian reconstruction experiment, not an end-to-end neural-network task. Only random rotations were compared; no learned orthogonal rotation, non-Gaussian activations, covariance drift, refresh schedule, gradient estimator, metadata/index cost, wall-clock speed, or CUDA path was tested. Monte Carlo evidence cannot establish the paper's universal inequality, and the paper text indicates the exact constant-one conjecture remains unresolved.

How to run: python3 kl_topk_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.15035) Aug 31, 2026

No significant difference vs. the tuned baseline (+1.3%, p=0.3883, wins 3 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.5059
Idea mean: 0.5127
Effect (Δ): 0.0068 (+1.3%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.3883 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±3.0%
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 9 configs

Limitations:

Only the registered sequence track and tiny transformer were tested. Learned rotations, covariance refresh, larger models, explicit activation-memory/index accounting, and wall-clock compression overhead were not tested.

How to run: python3 stage2_kl_topk_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

The KL-TopK PCA bottleneck was tested end-to-end in the registered sequence/transformer_tiny benchmark against the same transformer with coordinate top-k. PCA improved the standalone residual-energy sanity check, but task MSE was slightly worse and the paired permutation test was non-significant, so the idea did not produce the required benchmark win.

Baseline: Full paired baseline test MSE mean 0.5058855 (lr=0.004, keep=12).
Idea: Full paired PCA top-k test MSE mean 0.5126509 (lr=0.004, keep=12); delta +0.0067654, p=0.3883.
Math: PCA residual 1.910288 versus random-rotation residual 2.846760; formula implementation error 0.0.

Benchmark result