Replace an iid local or randomly sparse residual mixer with a distance-decaying long-range operator whose edge magnitudes are correlated through a shared latent Gaussian field. The paper predicts that these correlations qualitatively change low-energy spectral scaling and increase multiscale information propagation relative to iid long-range weights. Apply the operator as a spectrally normalized residual block so that the benefit comes from correlated scale coverage rather than uncontrolled…
Replace the usual JL dimension choice with a task-specific dimension budget based on the paper's asymptotic ranking law. For retrieval, nearest-neighbor search, or compressed attention keys, choose m from a target Kendall correlation and automatically expand the projection when measured ranking quality falls below the target.
Paper: Exact Limits of Random Projections for Preserving Geometry: Distance Recovery, Nearest-Neighbor Rankings, and Covariance Shape in Gaussian ModelsarXiv:2609.02155
Use a low-rank Tucker reconstruction as a structured backbone and quantize only its residual after an orthogonal rotation. The rotation preserves residual energy but redistributes it across coordinates, reducing dynamic-range imbalance and making 2- or 4-bit uniform quantization less damaging than direct quantization of the original KV tensor.
Paper: A JoLT for the KV cache: Near-lossless KV cache compression via joint Lagrangian allocation of Tucker ranks and a rotated residual for llmsarXiv:2607.12550
Replace selected dense neural-network operators by low-rank factors whose rank is selected by a randomized residual test at a user-specified tolerance. Construct candidate bases in large blocks for efficient matrix operations, then prune the block to the smallest rank that passes the residual criterion instead of treating the block size as the final rank.
Approximate a dense symmetric interaction matrix in a neural layer by \(\widehat A=C\widehat M C^{\top}\), but compute the small core \(\widehat M\) from a two-sided sketched least-squares fit rather than from the landmark principal submatrix. This preserves signed or indefinite directions and avoids exploding outputs caused by an almost-singular \(A(I,I)\).
Replace unconstrained predicted pairwise similarities with a correlation matrix whose diagonal is exactly one and whose spectrum is explicitly prevented from entering the nearly singular regime typical of high-dimensional elliptope samples. Add a soft spectral barrier during training and use a PSD-safe factorization at inference, so the model can represent dense correlations without relying on an unstable nearest-correlation-matrix repair.
Construct a symmetric token affinity matrix and approximate only its spectrally outlying token-mixing modes using a small set of sampled landmark columns. The resulting low-rank operator replaces an \(O(n^2)\) dense mixer by two skinny matrix multiplications, while the paper's residual guarantee predicts that large-magnitude global interaction modes are preserved.
Build a sparse graph by thresholding normalized token or item inner products, then use the leading eigenvectors of its centered adjacency matrix as geometric features or a low-rank attention-logit bias. The graph avoids storing all pairwise similarities, while the paper's spectral bound supplies a concrete signal-to-noise test for deciding whether the resulting embedding is trustworthy.
Regularize the eigenvalue spectrum of a neural representation or attention Gram matrix using the paper's universal-kernel spread-complexity curve. The loss penalizes spectral profiles that exhibit excessive level clustering or near-degeneracy, while allowing the desired amount of eigenvalue repulsion to be selected by a GOE-like, Poisson-like, or empirically calibrated target.
Use the paper's lower bound on nearest-correlation repair cost to detect when a neural network's pairwise similarity predictions contain too much globally incompatible off-diagonal energy. Instead of projecting every predicted matrix onto the correlation cone, train the network to reduce the repair-risk statistic or trigger expensive repair only when a cheap diagnostic predicts substantial distortion.
Use eigenvector delocalization as a mask-quality criterion rather than selecting a random sparse graph blindly. Penalize masks whose normalized adjacency has concentrated leading eigenvectors or disconnected or weakly connected components, while preserving the power-law distance prior. This creates a sparse routing graph that is less likely to trap information in local regions.
Calibrate the maximum attention logit in each head against the log-correlated extreme-value law instead of applying fixed clipping or a fixed max-norm penalty. Penalize only maxima that exceed the predicted log N minus three-quarter log log N baseline by an unusually large order-one fluctuation, allowing ordinary sharp attention while suppressing rare pathological spikes.