# KL-TopK Activation Bottleneck

- ID: 945
- Canonical URL: https://synthcore.org/idea/945/kl-topk-activation-bottleneck
- API JSON: https://synthcore.org/api/idea/945.json
- API Markdown: https://synthcore.org/api/idea/945.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.15035](https://arxiv.org/abs/2607.15035)
- Category: architecture
- Solves: memory, scalability, accuracy
- ML areas: embedding, mlp, quantization
- Math tags: linear-algebra, probability, convex-analysis, combinatorics
- Ratings: usefulness 7/10; difficulty 3/10; novelty 5/10

## Idea description

Construct a neural activation bottleneck by projecting hidden states into a fixed covariance-eigenbasis and retaining only the d largest-magnitude coordinates per sample. For Gaussian, decorrelated activations, the paper proves that adaptive top-d selection in the PCA basis has no greater expected residual energy than adaptive top-d selection after any other orthogonal rotation. This provides a principled alternative to learning an unrestricted rotation before sparsification.

## Mathematical statement

Let X=(X_1,...,X_p) be a centered Gaussian vector with independent coordinates and variances \lambda_1\geq\cdots\geq\lambda_p. Let z_{[1]}^2\geq\cdots\geq z_{[p]}^2 denote the decreasing rearrangement of the squared coordinates of z. For every orthogonal matrix U\in O(p) and every 1\leq d<p, the paper proves the correlation-gap inequality \mathbb{E}[\sum_{j=d+1}^{p}X_{[j]}^2]\leq\mathbb{E}[\sum_{j=d+1}^{p}(UX)_{[j]}^2]. The left-hand side is the expected squared residual after keeping the d largest coordinates of X; the right-hand side is the residual after first rotating X with U and then keeping the d largest rotated coordinates. Thus the KL basis is optimal for this adaptive nonlinear coordinate-retention objective in the stated Gaussian model. For neural activations h with covariance C, compute an eigendecomposition C=Q\operatorname{diag}(\lambda_1,...,\lambda_p)Q^    op with eigenvalues in descending order and use z=Q^    op h. The neural residual is \|z-\operatorname{TopD}(z)\|_2^2=\sum_{j=d+1}^{p}z_{[j]}^2. The paper also identifies the remaining comparison loss through the correlation gap of the rank-d uniform matroid and obtains a 1+O(1/\sqrt d) retained-energy comparison, meaning that any possible advantage of optimizing the orthogonal basis diminishes as d grows.

## Key formulas

- $$\mathbb{E}\sum_{j=d+1}^{p}X_{[j]}^{2}\leq\mathbb{E}\sum_{j=d+1}^{p}(UX)_{[j]}^{2},\qquad U\in O(p),\quad 1\leq d<p.$$
- $$\operatorname{TopD}(z)_i=z_i\,\mathbf{1}\{i\in I_d(z)\},\qquad I_d(z)\in\operatorname{arg\,max}_{|I|=d}\sum_{i\in I}|z_i|^2.$$
- $$\mathcal{L}_{\mathrm{top\text{-}d}}(h)=\left\|Q^\top h-\operatorname{TopD}(Q^\top h)\right\|_2^2=\sum_{j=d+1}^{p}(Q^\top h)_{[j]}^2,\qquad CQ=Q\operatorname{diag}(\lambda_1,\ldots,\lambda_p).$$
- $$\text{retained-energy comparison factor}=1+O\!\left(\frac{1}{\sqrt d}\right).$$

## Implementation notes

(1) Integration point: insert the module before an activation-compression interface, sparse autoencoder decoder, MLP bottleneck, or intermediate activation stored for gradient checkpointing. Let H\in\mathbb{R}^{B    imes p} be a batch of hidden states. Maintain a fixed orthogonal matrix Q and transform each row with Z=HQ. (2) Pseudocode: collect N calibration activations h_n; compute the mean \bar h and covariance C=(1/N)\sum_{n=1}^N(h_n-\bar h)(h_n-\bar h)^    op; compute Q and eigenvalues with an eigensolver; sort eigenvectors so \lambda_1\geq\cdots\geq\lambda_p. For each training or inference batch, compute Z=(H-\bar h)Q; for every row b, set I_b to the d indices with largest |Z_{b,i}|; form S_{b,i}=Z_{b,i} when i\in I_b and S_{b,i}=0 otherwise; reconstruct \widehat H=S Q^    op+\bar h if the downstream layer expects dense activations. Add \alpha\|H-\widehat H\|_2^2 to the task objective when the bottleneck is trainable. (3) Computed from the paper's mathematics: the orthogonal PCA transform, adaptive top-d rule, and exact residual-energy formula. Estimated empirically: covariance quality, deviation from Gaussianity, covariance drift during training, and the benefit of refreshing Q. In the first implementation, stop gradients through the top-d indices and keep Q fixed after calibration; a straight-through estimator can be tested later. (4) First cheap experiment: train a small Transformer on WikiText-2 or a two-layer MLP on CIFAR-10, then compare uncompressed activations, PCA-top-d, random-rotation-top-d, and a jointly learned orthogonal rotation at d/p values 0.1, 0.25, and 0.5. Measure reconstruction MSE, validation perplexity or accuracy, peak activation memory, and wall-clock time. The falsifiable target is that PCA-top-d matches or beats the learned-rotation baseline at equal d, while reducing stored activation values roughly in proportion to d/p; additionally, the performance gap between PCA and learned rotations should shrink as d increases.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a reproducible Gaussian KL-top-k activation bottleneck with calibration-time PCA, adaptive per-sample top-d retention, reconstruction, and numerical checks. The mechanism manifested: residual energy scaled linearly with covariance scale, isotropic inputs were rotation-invariant within Monte Carlo uncertainty, and PCA had lower residual than every tested random rotation across d=1..7. In the p=32 bottleneck, PCA retained 63.3%, 88.2%, and 97.8% of energy at d/p=0.094, 0.25, and 0.5, versus 39.8%, 70.6%, and 92.0% for a random rotation; the exact residual implementation error was 3.6e-15.

### Mechanism check

- Verdict: Built a reproducible Gaussian KL-top-k activation bottleneck with calibration-time PCA, adaptive per-sample top-d retention, reconstruction, and numerical checks. The mechanism manifested: residual energy scaled linearly with covariance scale, isotropic inputs were rotation-invariant within Monte Carlo uncertainty, and PCA had lower residual than every tested random rotation across d=1..7. In the p=32 bottleneck, PCA retained 63.3%, 88.2%, and 97.8% of energy at d/p=0.094, 0.25, and 0.5, versus 39.8%, 70.6%, and 92.0% for a random rotation; the exact residual implementation error was 3.6e-15.
- Confidence: 8/10
- Limitations: This is a synthetic Gaussian reconstruction experiment, not an end-to-end neural-network task. Only random rotations were compared; no learned orthogonal rotation, non-Gaussian activations, covariance drift, refresh schedule, gradient estimator, metadata/index cost, wall-clock speed, or CUDA path was tested. Monte Carlo evidence cannot establish the paper's universal inequality, and the paper text indicates the exact constant-one conjecture remains unresolved.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.3883
- Paired wins: 3/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/399/bench_report.json)
- [kl_topk_experiment.py](https://synthcore.org/code/399/kl_topk_experiment.py)
- [report.md](https://synthcore.org/code/399/report.md)
- [report_bench_2026-08-31T111740.md](https://synthcore.org/code/399/report_bench_2026-08-31T111740.md)
- [results.json](https://synthcore.org/code/399/results.json)
- [stage2_kl_topk_bench.py](https://synthcore.org/code/399/stage2_kl_topk_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/399)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.