Effective-Scale Sparse Projection
Source paper: Level-set entropy and sparse randomized embeddings arXiv:2607.23017 ⓘ · analyzed Sep 2, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace a dense token-mixing, MLP, or low-rank-adapter projection with a Bernoulli-signed sparse matrix normalized by the paper's predicted operator scale. Estimate the active representation dimension and use the effective scale to detect regimes in which extreme sparsity is likely to cause unstable amplification or dead rows.
Formulas
Mathematical statement
The paper defines r_* := max{r, log(k)/p}, where r is the dimension of a fixed input subspace, k is the output dimension, and p is the probability that a matrix entry is nonzero. It proves r <= r_* <= k and p r_* = max{p r, log(k)}. For Pi in R^{k x n} with entries Pi_ij = b_ij xi_ij, b_ij is Bernoulli(p), xi_ij is independent, mean zero, and bounded by one. For an isometry U_V from R^r into a fixed subspace V of R^n, the stated theorem gives ||Pi U_V|| <= C sqrt(k p) with high probability when k >= C r (log log r)^2 and p >= log(k)/k. We use W_hat = Pi/sqrt(kp), so the restricted operator norm should be O(1). This theorem controls only the largest singular value, not a lower singular-value guarantee, so the layer should be used for stable mixing or residual branches rather than blindly replacing an information-preserving embedding.
Implementation notes
Integrate this as a sparse linear projection in a residual token-mixing branch, a low-rank adapter, or an MLP expansion/compression layer. Let the input width be n, output width be k, and choose a target density p >= log(k)/k. Construct a fixed sparse mask b and independent Rademacher signs xi in {-1,+1}; initialize every nonzero weight as xi/sqrt(kp), and use a sparse matrix-multiplication kernel. Estimate the active rank r from a calibration matrix H of hidden activations using randomized SVD; choose the smallest r explaining 99 percent of the variance. Compute r_* = max(r, log(k)/p) and record whether k >= C r (log log(r+e))^2. Pseudocode: sample b and xi; set W = b elementwise-multiplied by xi divided by sqrt(kp); compute y = W h; optionally add y to a residual stream with a learned scalar gate initialized to 1/sqrt(2). The paper supplies the normalization and effective-scale diagnostic; rank estimation and the constant C are empirical. First test a width-512 two-layer Transformer on WikiText-2, replacing one dense projection by W. Compare dense, unnormalized sparse, and normalized sparse models at equal training steps, parameters, and measured sparse FLOPs. Pre-register that normalized sparse weights keep the 99th-percentile ratio ||Wh||/||h|| below 2.0, that this ratio is stable within 10 percent when p changes at fixed kp, and that p below log(k)/k increases upper-tail amplification or dead-row frequency. Remove only the 1/sqrt(kp) factor for the mechanism ablation. Falsify the transfer if operator-norm tails and gradient spikes do not decrease, or if any loss improvement disappears after equalizing actual FLOPs.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.