△ Mechanism confirmed, baseline not beaten
2026
Replace a square dense projection in a Transformer or MLP with a trainable recursive butterfly matrix. The layer preserves multiscale channel interactions while constraining every complementary row-column block to rank at most k, reducing parameters and enabling recursive structured matrix-vector products. Unlike an arbitrary sparse layer, the construction has an explicit recursive factorization and a quasi-optimal approximation guarantee among matrices with the same butterfly rank.
Useful8/10
Difficulty6/10
Novelty5/10
✓✓ Beats tuned baseline
2026
Replace dense self-attention by a sparse attention graph whose neighborhoods satisfy the paper's size-dependent expansion condition. This preserves a logarithmically controlled route for every token subset to communicate with the rest of the sequence, reducing quadratic attention cost without allowing disconnected or poorly mixed token groups.
Useful8/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Implement causal linear attention in chunks and combine chunk summaries with an associative scan instead of carrying the recurrent state through all chunks sequentially. This preserves the exact causal computation while reducing inter-chunk dependency depth from the number of chunks to its logarithm, enabling substantially more GPU parallelism for long-context training and prefill.
Useful8/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a dense directed attention matrix by a collection of K learned source-to-hub-to-target interactions. Each hub corresponds to a directed biclique, allowing many source tokens to communicate with many target tokens using O(NK) rather than O(N^2) pair interactions. The construction preserves asymmetric information flow and can be initialized from a graph cover of high-attention edges.
Useful8/10
Difficulty6/10
Novelty6/10
✗ Mechanism failed
2026
Use a spherical-design codebook and the paper's polar slack factorization to create a nonnegative geometric interaction bias for attention or expert routing. The resulting kernel is generated by a rank-one term and a rank-at-most-d term, and entries close to zero can define a structured sparse mask instead of relying only on learned top-k selection.
Useful7/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace dense or single-dilation sparse attention with two sequential sparse attention stages whose offsets form co-prime arithmetic progressions. The first stage mixes tokens separated by multiples of M2, the second by multiples of M1; their composition reaches virtual offsets mM2+nM1, providing many structured long-range interactions from only M1+M2-1 physical offset families. Use causal masking and residual connections so the module can replace a standard transformer attention block without…
Useful7/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a quadratic pairwise attention or graph aggregation kernel with a compact, translation-invariant indefinite kernel approximated by signed random Fourier features. The feature map preserves the kernel's negative spectral mass through a diagonal sign matrix, so the resulting linear-time aggregation can represent similarities that ordinary positive-definite random features cannot.
Useful7/10
Difficulty5/10
Novelty7/10
✓✓ Beats tuned baseline
2026
Replace dense graph self-attention with two parallel branches: exact softmax attention only over graph neighbors and a global linear-attention branch that summarizes all nodes through feature-space statistics. A learned node-wise gate interpolates between the branches, allowing locally structured nodes to use sparse attention while retaining a global-information path.
Useful7/10
Difficulty4/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Construct a cheap graph or token-mixing operator by partitioning nodes into k blocks using the bottom nonconstant eigenvectors of P squared, then replacing dense pairwise mixing with conditional averaging inside each block followed by one baseline propagation step. Unlike ordinary spectral clustering, the bottom modes target partitions where block labels are rapidly destroyed by P, producing an aggressively mixing representation layer rather than a community-preserving pooling layer. The…
Useful7/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace an arbitrary graph-attention mask with a fractional edge mask lying in the intersection of the spanning-tree polytope and twice the matching polytope. The mask represents a distribution over connected spanning trees while imposing expected degree at most two at every vertex, after which sampled trees can be used for sparse message passing.
Useful7/10
Difficulty7/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace full PSD self-attention with a pivoted Cholesky/Nyström approximation whose landmarks are sampled from the unexplained diagonal mass. Tokens with large residual self-similarity are more likely to become landmarks, so the rank budget is spent on difficult regions rather than uniformly selected tokens.
Useful7/10
Difficulty5/10
Novelty5/10
✗ Failed on benchmark
2026
Insert a linear Johnson–Lindenstrauss bottleneck around a set of jointly processed representations, choosing its width from the sharp finite-set dimension bound rather than from the model's nominal hidden size. The projection should preserve pairwise distances between tokens, patches, or retrieved items, allowing a downstream attention or MLP block to operate at lower width while retaining the geometry relevant to similarity computations.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Replace multiplicative query-key attention scores with an affinity based on the l1 distance between first-spike latency vectors. For each query token and key token, small latency differences produce large affinity and distant timings decay exponentially, yielding a locality-sensitive attention pattern naturally compatible with leaky spiking neurons.
Useful7/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace full agent-to-agent state transmission with coefficients in a learned dominant Koopman mode basis. Agents communicate only the leading spectral coordinates that explain slowly decaying collective behavior, while retaining a certificate based on the spectral gap and subdominant eigenvalue to decide whether the compressed representation is safe.
Useful7/10
Difficulty6/10
Novelty6/10
✓✓ Beats tuned baseline
2026
Replace dense token-to-token attention by a multiscale spiderweb communication pattern. Tokens first aggregate upward through a dyadic hierarchy, communicate horizontally only with a small number of cells at the appropriate height, and then receive information broadcast downward. Hyperbolic distance supplies a principled rule for choosing the height at which two tokens interact: nearby tokens interact at fine scales, while far-apart tokens interact through coarse representatives.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Use the relaxed QFT tensor-network topology as a trainable norm-preserving mixer inside a neural block, replacing a dense token-mixing matrix or an expensive global convolution. The network learns data-adapted global interactions while retaining structured O(N log^2 N) application and an exact cheap inverse, making it suitable for image tokens, long sequences, or reversible residual blocks.
Useful7/10
Difficulty6/10
Novelty6/10
✓✓ Beats tuned baseline
2026
Replace a dense mixing or attention matrix on tokens arranged on a Cartesian grid by a product of learned or fixed one-dimensional concentration operators. The layer applies one axis operator at a time, reducing parameter and compute cost while enforcing that the global operator is a positive contraction with controlled spectral leakage.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Replace dense attention between tokens on opposite sides of a one-dimensional boundary or segment split with a dyadic low-rank approximation of a Cauchy/Hankel distance kernel. Each distance-scale block uses O(log(1/\varepsilon)) features, and the number of active scales grows only logarithmically with context length after discarding a narrow boundary layer. This is especially suitable for a relative-position attention branch or state-space-like long-range branch, rather than arbitrary…
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a portion of quadratic key-value attention or an external episodic table with a per-sample matrix fast memory updated by rank-one delta corrections. The memory directly learns a linear key-to-value map and can be carried across sequence segments, providing cheap online adaptation with constant state size per head.
Useful7/10
Difficulty5/10
Novelty5/10
✗ Failed on benchmark
2026
Construct overlapping attention windows from a block design instead of using one dense sequence-by-sequence attention matrix. Every token appears in exactly $r$ windows and every token pair co-occurs in exactly $\lambda$ windows, giving uniform coverage and avoiding the uneven connectivity of arbitrary sparse masks.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace dense query-key attention with an adaptive cross approximation constructed from selected query and key pivot tokens. At each rank, choose the pivot pair that removes large estimated residual energy, update the residual by a rank-1 cross correction, and stop when the residual estimate reaches a target tolerance. The resulting factorization computes approximate attention using a small number of landmark interactions while adapting to the actual token distribution.
Useful7/10
Difficulty6/10
Novelty5/10
✗ Mechanism failed
2026
Replace dense self-attention by a sparse mask on a one-dimensional or ordered token geometry, retaining local neighbors and adding long-range edges with probability proportional to distance raised to \(-(1+\sigma)\). Use \(\sigma\approx0.8\text{--}0.85\) as the initial regime because the paper finds that this range supports delocalized, GOE-like connectivity despite sparse bonds. The resulting layer has linear or near-linear attention cost while maintaining long-range paths.
Useful7/10
Difficulty4/10
Novelty6/10
✗ Failed on benchmark
2026
Partition tokens into a rectangular grid and use Latin-square labels to define several sparse attention heads. Each head attends only to tokens sharing a row, column, or Latin label, while orthogonality guarantees that every pair of labels occurs at most once, reducing systematic blind spots and repeated collisions. The resulting masks are deterministic, reusable across examples, and can be generated without learned routing scores.
Useful7/10
Difficulty5/10
Novelty7/10
✗ Failed on benchmark
2026
Replace quadratic self-attention over a sequence with a bank of K auxiliary exponentially decaying states whose rates are fitted directly from the empirical autocorrelation of the sequence features. Each mode captures a distinct time scale, so the module can represent short- and long-range dependencies with O(TK) computation and O(K) recurrent memory rather than storing all previous tokens. Constrain decay rates to be positive and use the paper's extended Markovian block structure to obtain a…
Useful7/10
Difficulty5/10
Novelty4/10