Mechanism confirmed, baseline not beaten
2026
Log-Depth Chunked Linear-Attention Scan
Implement causal linear attention in chunks and combine chunk summaries with an associative scan instead of carrying the recurrent state through all chunks sequentially. This preserves the exact causal computation while reducing inter-chunk dependency depth from the number of chunks to its logarithm, enabling substantially more GPU parallelism for long-context training and prefill.
Paper: Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
arXiv:2607.17419