# Prolate Energy-Preserving Bottleneck

- ID: 2368
- Canonical URL: https://synthcore.org/idea/2368/prolate-energy-preserving-bottleneck
- API JSON: https://synthcore.org/api/idea/2368.json
- API Markdown: https://synthcore.org/api/idea/2368.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.15808](https://arxiv.org/abs/2608.15808)
- Category: memory
- Solves: memory, inference-speedup, accuracy
- ML areas: memory, inference-speedup, transformer, cnn
- Math tags: spectral-theory, harmonic-analysis, linear-algebra, approximation-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Insert a fixed DPSS/prolate projection before an expensive neural block, retaining exactly the modes whose time-frequency concentration eigenvalues exceed a target threshold. Use the paper's tail-quantile formula to choose the projection rank from sequence length, effective bandwidth, and tolerated energy loss, then optionally learn a small correction in the retained coordinates. Unlike a Fourier truncation, the basis is optimized for simultaneous localization in the finite input window and the assumed frequency band.

## Mathematical statement

The concentration operator is $S_c=PQP$ on $L^2(0,c)$, where $P$ restricts a function to the time interval $(0,c)$ and $Q$ projects its Fourier transform onto $(-1/2,1/2)$. Its eigenvalues satisfy $1>\lambda_0(c)\geq\lambda_1(c)\geq\cdots>0$, and $N_a(c)=\#\{n:\lambda_n(c)>a\}$. For a discarded-energy threshold $\delta\in(0,1/2)$ and $\bar L=\log((1-\delta)/\delta)$, the paper gives $N_\delta(c)=c+\pi^{-2}\bar L\log(4\pi^2c/\bar L)+O_A(\log c+\bar L)$ uniformly for $6\leq\bar L\leq A\log c$. Here $c$ is the time-bandwidth scaling parameter, $\delta$ is the eigenvalue threshold, $\bar L$ is its log-odds, and $A$ is a fixed uniformity constant. The underlying determinant estimate is $\log\det(I+(e^{2\omega}-1)K_s)=4\omega s/\pi+2\pi^{-2}\omega^2\log(4s)+2\log|G(1+i\omega/\pi)|^2+O_A((1+\omega)^4\log^2s/s)$ for $0\leq\omega\leq A\log s$, where $K_s$ is the sine-kernel operator, $\omega$ is the growing determinant parameter, $s$ is the kernel scale, and $G$ is the Barnes $G$-function. In the neural adaptation, form a finite concentration matrix $C$, eigendecompose it as $C=U\operatorname{diag}(\hat\lambda_i)U^\top$, and retain $r=\#\{i:\hat\lambda_i>\delta\}$ modes.

## Key formulas

- $$S_c=PQP,\qquad N_a(c)=\#\{n:\lambda_n(c)>a\},\qquad 1>\lambda_0(c)\geq\lambda_1(c)\geq\cdots>0.$$
- $$N_\delta(c)=c+\pi^{-2}\bar L\log\!\left(\frac{4\pi^2c}{\bar L}\right)+O_A(\log c+\bar L),\qquad \bar L=\log\frac{1-\delta}{\delta},\quad 6\leq\bar L\leq A\log c.$$
- $$\log\det\!\left(I+(e^{2\omega}-1)K_s\right)=\frac{4\omega s}{\pi}+\frac{2\omega^2}{\pi^2}\log(4s)+2\log\left|G\!\left(1+\frac{i\omega}{\pi}\right)\right|^2+O_A\!\left(\frac{(1+\omega)^4\log^2s}{s}\right).$$
- $$C_{ij}=\begin{cases}\dfrac{\sin(2\pi W(i-j))}{\pi(i-j)},&i\neq j,\\2W,&i=j,\end{cases}\qquad C=U\operatorname{diag}(\hat\lambda_1,\ldots,\hat\lambda_T)U^\top,\qquad Z=XU_{[:,1:r]},\quad \hat X=ZU_{[:,1:r]}^\top.$$

## Implementation notes

Integrate this at the input of an expensive sequence module. For a tensor X with shape batch by sequence length T by channels D, apply the same orthonormal DPSS matrix U along the sequence axis before attention, a convolutional stack, or an MLP. Choose an effective normalized bandwidth W in (0,0.5), construct the symmetric concentration matrix C using the sinc formula above, and compute its eigenvectors with a symmetric eigensolver. Sort eigenvalues descending and retain r modes. Set delta to the maximum acceptable discarded concentration, compute barL=log((1-delta)/delta), and use r_asym=c + barL/pi^2 * log(4*pi^2*c/barL) as an initial rank estimate; because discrete normalization differs from the continuous theorem, clip r_asym to [1,T] and verify it using the empirical eigenvalues. Pseudocode: construct C; eigvals,eigvecs=eigh(C); sort descending; r=ceil(r_asym); Z=einsum('btd,tr->brd',X,U[:,:r]); Y=expensive_block(Z); output=einsum('brd,tr->btd',Y,U[:,:r].T), optionally adding a residual bypass for discarded coordinates. The theorem supplies the rank schedule; the eigendecomposition and actual reconstruction error are computed numerically. Start with a small Transformer on WikiText-103 or Long Range Arena at T=1024, comparing full attention, Fourier truncation, random orthogonal projection, and a learned linear bottleneck. Measure validation perplexity, reconstruction error, peak activation memory, and attention FLOPs. Success means at least 25 percent lower attention memory or FLOPs with no more than 1 percent relative perplexity degradation, and a better rank-versus-error curve than Fourier truncation on locally concentrated signals.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a fixed prolate/DPSS bottleneck with theorem-inspired rank selection, tensor encode/decode, Fourier comparison, and numerical verification. The concentration matrix had eigenvalues in [0,1] up to numerical precision; ceil(asymptotic rank)=44 exactly matched the empirical count above delta=0.1. On band-limited covariance samples at rank 44, DPSS reconstruction error was 0.00108 versus 0.00885 for Fourier and 0.535 for a random subspace, while reducing quadratic attention work by 79.0%. This is a clear toy-signal win, but no trained neural block or language-model perplexity experiment was run.

### Mechanism check

- Verdict: Built a fixed prolate/DPSS bottleneck with theorem-inspired rank selection, tensor encode/decode, Fourier comparison, and numerical verification. The concentration matrix had eigenvalues in [0,1] up to numerical precision; ceil(asymptotic rank)=44 exactly matched the empirical count above delta=0.1. On band-limited covariance samples at rank 44, DPSS reconstruction error was 0.00108 versus 0.00885 for Fourier and 0.535 for a random subspace, while reducing quadratic attention work by 79.0%. This is a clear toy-signal win, but no trained neural block or language-model perplexity experiment was run.
- Confidence: 8/10
- Limitations: Only a small NumPy toy experiment on synthetic samples from the assumed band-limited covariance was tested. No Transformer, attention memory measurement, wall-clock speed benchmark, learned correction, residual bypass, WikiText/LRA data, or perplexity/accuracy comparison was run; the asymptotic theorem was checked through its rank prediction rather than independently reproving its error bound.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_prolate.py](https://synthcore.org/code/929/bench_prolate.py)
- [bench_report.json](https://synthcore.org/code/929/bench_report.json)
- [experiment.py](https://synthcore.org/code/929/experiment.py)
- [prolate_bottleneck.py](https://synthcore.org/code/929/prolate_bottleneck.py)
- [report.md](https://synthcore.org/code/929/report.md)
- [report_bench_2026-09-02T162721.md](https://synthcore.org/code/929/report_bench_2026-09-02T162721.md)
- [results.json](https://synthcore.org/code/929/results.json)
- [validate.py](https://synthcore.org/code/929/validate.py)
- [validation_results.json](https://synthcore.org/code/929/validation_results.json)
- [Download all files as ZIP](https://synthcore.org/download/929)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.