Beats tuned baseline
2026
Sublinear-expander sparse attention
Replace dense self-attention by a sparse attention graph whose neighborhoods satisfy the paper's size-dependent expansion condition. This preserves a logarithmically controlled route for every token subset to communicate with the rest of the sequence, reducing quadratic attention cost without allowing disconnected or poorly mixed token groups.
Paper: Recent progress in graph theory using expansion
arXiv:2607.26049