arXiv:2607.23017 2026 Architecture 2 ideas extracted · analyzed Sep 2, 2026
The paper develops a constructive method for controlling sparse random operators on low-dimensional subspaces by decomposing vectors into dyadic coordinate level sets and separating heavy (Tall) and diffuse (Flat) contributions. The most transferable asset is the effective scale r_* = max{r, log(k)/p}, which identifies when sparsity rather than intrinsic dimension controls fluctuations, together with the normalization ||Pi U_V|| = O(sqrt(kp)). A practical neural-network transfer is a spectrally calibrated sparse projection or token-mixing layer whose density and initialization are chosen from the activation rank and sparsity. A second experiment can use magnitude bands to allocate connectivity unequally, protecting large coordinates while using randomized connectivity for diffuse coordinates.
Replace a dense token-mixing, MLP, or low-rank-adapter projection with a Bernoulli-signed sparse matrix normalized by the paper's predicted operator scale. Estimate the active representation dimension and use the effective scale to detect regimes in which extreme sparsity is likely to cause unstable amplification or dead rows.
Partition activations into dyadic magnitude bands and allocate sparse connectivity separately to heavy and diffuse coordinates. Protect high-magnitude coordinates with more reliable connections while using randomized flat connectivity for the many small coordinates, keeping the total number of nonzeros fixed.