arXiv:2607.26049 2026 Architecture 2 ideas extracted · analyzed Aug 31, 2026
The paper supplies a concrete connectivity primitive for sparse computation graphs: every moderately sized subset of nodes must have a proportionally large external neighborhood. This is directly transferable to sparse attention and token-to-expert routing, where arbitrary sparsification can create isolated groups and prevent information or gradients from mixing. The especially useful feature is the sublinear expansion guarantee, which weakens expansion only logarithmically with subset size and therefore remains meaningful for sparse graphs with nonuniform local structure. The best first transfer is to construct or learn sparse connectivity subject to an expansion constraint, then measure mixing, loss descent, and quality at fixed edge count.
Replace dense self-attention by a sparse attention graph whose neighborhoods satisfy the paper's size-dependent expansion condition. This preserves a logarithmically controlled route for every token subset to communicate with the rest of the sequence, reducing quadratic attention cost without allowing disconnected or poorly mixed token groups.
Treat token-to-expert assignments as a bipartite routing graph and penalize patterns that leave token groups sharing too few experts. Enforcing sublinear expansion makes routing diversity explicit: even a moderately sized group of tokens must reach many distinct experts, reducing expert collapse and improving load balance without requiring every token to use every expert.