# Sharp JL Hidden-State Bottleneck

- ID: 2261
- Canonical URL: https://synthcore.org/idea/2261/sharp-jl-hidden-state-bottleneck
- API JSON: https://synthcore.org/api/idea/2261.json
- API Markdown: https://synthcore.org/api/idea/2261.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.13782](https://arxiv.org/abs/2608.13782)
- Category: architecture
- Solves: memory, speedup, accuracy
- ML areas: transformer, attention, mlp, embedding, inference-speedup
- Math tags: linear-algebra, metric-geometry, probability, approximation-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Insert a linear Johnson–Lindenstrauss bottleneck around a set of jointly processed representations, choosing its width from the sharp finite-set dimension bound rather than from the model's nominal hidden size. The projection should preserve pairwise distances between tokens, patches, or retrieved items, allowing a downstream attention or MLP block to operate at lower width while retaining the geometry relevant to similarity computations.

## Mathematical statement

The paper defines distortion at most 1+epsilon by requiring a scale lambda>0 such that lambda ||x-y||_2 <= ||f(x)-f(y)||_2 <= (1+epsilon) lambda ||x-y||_2 for every pair x,y in an n-point set X subset R^d. Its sharp dimension bound is r = Theta(min{d, n-1, log(2+epsilon^2 n)/epsilon^2}), and the upper bound is attained by a linear map. Here x and y are input representations, d is their original feature dimension, n is the number of representations whose mutual geometry must be preserved, epsilon is the allowed relative distortion, and r is the projected dimension. Use a Gaussian linear map P in R^{r x d}, with P_ij sampled independently from N(0,1/r), so that f(x)=Px approximately preserves Euclidean norms and pairwise distances. The theorem supplies the functional width rule, while the hidden absolute constant in Theta must be selected empirically.

## Key formulas

- $$\lambda\left\lVert x-y\right\rVert_{2}\leq\left\lVert f(x)-f(y)\right\rVert_{2}\leq(1+\varepsilon)\lambda\left\lVert x-y\right\rVert_{2},\qquad \forall x,y\in X.$$
- $$r=\Theta\left(\min\left\{d,n-1,\frac{\log(2+\varepsilon^{2}n)}{\varepsilon^{2}}\right\}\right).$$
- $$P_{ij}\sim\mathcal N(0,1/r),\qquad z_i=Px_i,\qquad \widehat d_{ij}=\lVert z_i-z_j\rVert_2.$$

## Implementation notes

(1) Integration point: place a frozen or trainable projection P immediately before a width-sensitive block, such as self-attention or an MLP, for a tensor X with shape [batch,n,d]. In the first MVP, use a frozen Gaussian P and make the downstream block operate at width r. Compute r = ceil(C * min(d, n-1, log(2 + epsilon^2*n)/epsilon^2)), with C in {1,2,4} because the theorem hides constants, and cap r at d. (2) Pseudocode: sample P[i,j] from Normal(0,1/r); compute Z = X @ P.T; run the compressed attention or MLP on Z; optionally reconstruct Y = Z @ Q.T if the following block requires width d. During training, add L_dist = mean over sampled pairs of (||Z_i-Z_j||_2/(||X_i-X_j||_2+delta)-1)^2, with a small delta for numerical stability. (3) The mathematical calculation is the width schedule and target distortion definition. Estimate empirically the distortion ratio rho_ij = ||Z_i-Z_j||_2/(||X_i-X_j||_2+delta), its 95th percentile, downstream task loss, and activation memory. Compare frozen Gaussian, orthogonal-row, and learned projections. (4) First experiment: use a small Transformer on CIFAR-10 patch tokens or a two-layer language model on WikiText-2. Compare the original width against epsilon values 0.1, 0.2, and 0.3 and C values 1, 2, and 4. Measure validation accuracy or perplexity, pairwise-distance distortion, activation memory, and throughput. Success means at least 1.5x lower activation cost with less than 1% relative accuracy loss, and a transition in useful width near the predicted min of d, n-1, and epsilon^{-2} log(2+epsilon^2 n).

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built and verified a frozen Gaussian JL hidden-state bottleneck with the sharp finite-set width rule. The core predictions were observed: mean squared norm ratio 0.986 versus 1, fixed-pair ratio SD 0.1235 versus the predicted 0.125 at r=32, and finite-set distortion decreased approximately with the predicted r^{-1/2} trend; the schedule also saturated at n−1 and d as expected. In the toy downstream task, projected representations preserved 100% validation accuracy at r=16–23 versus full width d=96, giving 4.2–6x lower activation width, but no realistic Transformer or speed benchmark was tested.

### Mechanism check

- Verdict: Built and verified a frozen Gaussian JL hidden-state bottleneck with the sharp finite-set width rule. The core predictions were observed: mean squared norm ratio 0.986 versus 1, fixed-pair ratio SD 0.1235 versus the predicted 0.125 at r=32, and finite-set distortion decreased approximately with the predicted r^{-1/2} trend; the schedule also saturated at n−1 and d as expected. In the toy downstream task, projected representations preserved 100% validation accuracy at r=16–23 versus full width d=96, giving 4.2–6x lower activation width, but no realistic Transformer or speed benchmark was tested.
- Confidence: 8/10
- Limitations: The downstream task is synthetic and linearly separable, so accuracy is not a stringent test of representation quality. No CIFAR-10/LM Transformer, wall-clock throughput, actual attention/MLP memory, learned or orthogonal projections, repeated seeds, or rigorous high-probability distortion certification was evaluated; the finite-set maximum-error scaling comparison is a heuristic concentration-scale check rather than a theorem-tight constant test.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.66785
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/893/bench_report.json)
- [report.md](https://synthcore.org/code/893/report.md)
- [report_bench_2026-09-01T135120.md](https://synthcore.org/code/893/report_bench_2026-09-01T135120.md)
- [results.json](https://synthcore.org/code/893/results.json)
- [run_experiment.py](https://synthcore.org/code/893/run_experiment.py)
- [stage2_jl_bench.py](https://synthcore.org/code/893/stage2_jl_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/893)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.