Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones
arXiv:2607.17419
2026
Architecture
2 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper turns linear-attention feature-map design into a geometric problem: attention weights remain nonnegative when the feature map takes values in a self-dual cone, and rank-one PSD features provide a structured construction. The transferable asset is a finite-dimensional kernel whose associative-recall interference can be analyzed through spherical-packing and Welch bounds, rather than choosing random positive features heuristically. A second practical contribution is an associative chunk-state formulation that replaces the sequential inter-chunk recurrence with a logarithmic-depth scan, preserving exact linear-time inference while exposing substantial GPU parallelism.
Ideas from this paper
△ Mechanism confirmed, baseline not beaten
2026
Implement causal linear attention in chunks and combine chunk summaries with an associative scan instead of carrying the recurrent state through all chunks sequentially. This preserves the exact causal computation while reducing inter-chunk dependency depth from the number of chunks to its logarithm, enabling substantially more GPU parallelism for long-context training and prefill.
Useful8/10
Difficulty6/10
Novelty6/10
✗ Failed on benchmark
2026
Replace the usual random or elementwise-positive linear-attention feature map with a rank-one positive-semidefinite feature map derived from query and key vectors. For normalized inputs, the resulting kernel is the squared inner product, which is nonnegative and gives a geometrically structured interference pattern that is better suited to associative recall than an arbitrary low-rank feature map.
Useful8/10
Difficulty6/10
Novelty7/10