Spectral latent geometry for sparse attention

(1) Integrate this at the input of a transformer, sparse-attention block, or graph neural network. For token or item vectors X in R^(n by d), normalize each row as U_i = X_i/(||X_i||_2 + epsilon). Construct a symmetric sparse graph using A_ij = 1[U_i^T U_j >= tau] for i != j and A_ii = 0. The threshold tau should be selected from a sampled empirical quantile so that approximately a target fraction p of pairs are retained. Use p between 4 log(n)/n and 0.05 for the first sweep. (2) Form the centered operator C(v) = A v - p((1^T v)1 - v), without materializing the dense matrix pJ. Run randomized Lanczos or subspace iteration for r eigenvectors, producing V_r, and set Z = sqrt(n)V_r. Concatenate Z with X before the network, or add the low-rank bias B = alpha Z Z^T/sqrt(r) to sparse-attention logits. (3) Compute sigma_hat = sqrt(n p log n) + n p tau from the paper's bound. Estimate the smallest retained eigenvalue lambda_hat_r and gate the feature when lambda_hat_r <= kappa sigma_hat. The theorem-derived quantities are the threshold graph, centering, spectral noise proxy, and density diagnostic; the useful rank r, scale alpha, and kappa are tuned on validation data. (4) First test on a small transformer for CIFAR-100 patches or a Cora graph, comparing dense attention, standard top-k similarity sparsification, and this method at matched average degree. Record validation accuracy, peak memory, graph-build time, eigen-solver time, spectral ratio lambda_hat_r/sigma_hat, and loss at equal wall-clock time. Success is comparable or better accuracy with lower memory and near-linear sparse aggregation cost; deliberately lowering p below the connectivity scale should cause the predicted spectral collapse.