The Sharp Dimension Bound in the Johnson--Lindenstrauss Lemma
arXiv:2608.13782
2026
Architecture
2 ideas extracted · analyzed Sep 1, 2026
What the math gives to ML
The paper establishes the sharp target dimension for preserving all pairwise Euclidean distances among an n-point set, including finite-sample and ambient-dimension regimes hidden by the usual O(epsilon^{-2} log n) statement. The transferable asset is a principled feature-width rule: a linear projection need not use more than the smaller of the intrinsic ambient dimension, n-1, and log(2+epsilon^2 n)/epsilon^2, while preserving geometry up to 1+epsilon distortion. This suggests replacing uniformly wide hidden representations with calibrated linear bottlenecks whose width is selected from the number of objects being jointly processed and the tolerated distortion. Because the extraction gives no explicit optimized projection beyond the standard Gaussian example, the theorem should be tested as a width-selection hypothesis using randomized or learned projections.
Ideas from this paper
✗ Failed on benchmark
2026
Insert a linear Johnson–Lindenstrauss bottleneck around a set of jointly processed representations, choosing its width from the sharp finite-set dimension bound rather than from the model's nominal hidden size. The projection should preserve pairwise distances between tokens, patches, or retrieved items, allowing a downstream attention or MLP block to operate at lower width while retaining the geometry relevant to similarity computations.
Useful7/10
Difficulty5/10
Novelty5/10
Unverified
2026
Use the number of tokens participating in an attention operation to adapt the projected query/key dimension according to the sharp Johnson–Lindenstrauss bound. Short attention windows receive a dimension limited by n-1, while longer windows use a logarithmic dimension, avoiding a fixed large head dimension when the current attention set contains too few points to require it.
Useful6/10
Difficulty6/10
Novelty6/10