The Sharp Dimension Bound in the Johnson--Lindenstrauss Lemma

arXiv:2608.13782 2026 Architecture 2 ideas extracted · analyzed Sep 1, 2026

What the math gives to ML

The paper establishes the sharp target dimension for preserving all pairwise Euclidean distances among an n-point set, including finite-sample and ambient-dimension regimes hidden by the usual O(epsilon^{-2} log n) statement. The transferable asset is a principled feature-width rule: a linear projection need not use more than the smaller of the intrinsic ambient dimension, n-1, and log(2+epsilon^2 n)/epsilon^2, while preserving geometry up to 1+epsilon distortion. This suggests replacing uniformly wide hidden representations with calibrated linear bottlenecks whose width is selected from the number of objects being jointly processed and the tolerated distortion. Because the extraction gives no explicit optimized projection beyond the standard Gaussian example, the theorem should be tested as a width-selection hypothesis using randomized or learned projections.

Ideas from this paper

Failed on benchmark 2026

Sharp JL Hidden-State Bottleneck

Insert a linear Johnson–Lindenstrauss bottleneck around a set of jointly processed representations, choosing its width from the sharp finite-set dimension bound rather than from the model's nominal hidden size. The projection should preserve pairwise distances between tokens, patches, or retrieved items, allowing a downstream attention or MLP block to operate at lower width while retaining the geometry relevant to similarity computations.

Useful7/10
Difficulty5/10
Novelty5/10
Paper: The Sharp Dimension Bound in the Johnson--Lindenstrauss Lemma arXiv:2608.13782
Unverified 2026

Token-Count-Aware Attention Projection

Use the number of tokens participating in an attention operation to adapt the projected query/key dimension according to the sharp Johnson–Lindenstrauss bound. Short attention windows receive a dimension limited by n-1, while longer windows use a logarithmic dimension, avoiding a fixed large head dimension when the current attention set contains too few points to require it.

Useful6/10
Difficulty6/10
Novelty6/10
Paper: The Sharp Dimension Bound in the Johnson--Lindenstrauss Lemma arXiv:2608.13782