△ Mechanism confirmed, baseline not beaten
2026
Allocate different entropy budgets to different KV-cache blocks instead of assigning every token and head the same nominal bitwidth. Use the ECASQ Lagrangian so high-variance or attention-sensitive blocks receive more codepoints, while predictable blocks collapse to fewer symbols and become highly compressible. Preserve unbiasedness per scalar or block so reconstructed keys and values have zero mean quantization error conditional on the original tensor.
Useful8/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace fixed-grid stochastic quantization of each tensor block with an adaptive ordered codebook selected under both an entropy budget and a maximum number of codepoints. Within every interval between adjacent codepoints, use unbiased stochastic interpolation, so the quantized block remains unbiased while the emitted symbol distribution becomes easier for arithmetic or Huffman coding to compress. The representation should reduce actual compressed bytes at fixed MSE, or reduce MSE at fixed…
Useful8/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Turn an iterative optimization or equilibrium computation inside a neural network into a differentiable layer whose backward pass solves the implicit adjoint system with conjugate gradients or GMRES using only automatic-differentiation matrix-vector products. This avoids storing unrolled iterations and avoids explicit Hessian or Jacobian construction, enabling longer solver horizons and lower-memory implicit architectures.
Useful8/10
Difficulty6/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a square dense projection in a Transformer or MLP with a trainable recursive butterfly matrix. The layer preserves multiscale channel interactions while constraining every complementary row-column block to rank at most k, reducing parameters and enabling recursive structured matrix-vector products. Unlike an arbitrary sparse layer, the construction has an explicit recursive factorization and a quasi-optimal approximation guarantee among matrices with the same butterfly rank.
Useful8/10
Difficulty6/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a dense neural-network weight tensor with a graph tensor network whose physical modes and internal edge ranks are specified by a sparse rank-adjacency matrix. Unlike tensor-train or hierarchical Tucker layers, the graph can contain selected cycles and skip connections between tensor modes, allowing the factorization topology to match correlations in the weight tensor. Fit the layer with GTN-SVD at a prescribed tolerance and compare accuracy, parameter count, and tensor-contraction…
Useful8/10
Difficulty6/10
Novelty5/10
✓✓ Beats tuned baseline
2026
Replace dense self-attention by a sparse attention graph whose neighborhoods satisfy the paper's size-dependent expansion condition. This preserves a logarithmically controlled route for every token subset to communicate with the rest of the sequence, reducing quadratic attention cost without allowing disconnected or poorly mixed token groups.
Useful8/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Before quantizing a matrix product, reparameterize its factors as A'=AT and B'=T^{-1}B, preserving the exact full-precision product while changing the quantization difficulty of each factor. Choose a positive diagonal T=diag(t_1,...,t_K) that minimizes predicted post-quantization product error, rather than using output-channel scaling or a fixed heuristic grid. The gauge can be shared across several products when transformed-copy cost matters.
Useful8/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Implement causal linear attention in chunks and combine chunk summaries with an associative scan instead of carrying the recurrent state through all chunks sequentially. This preserves the exact causal computation while reducing inter-chunk dependency depth from the number of chunks to its logarithm, enabling substantially more GPU parallelism for long-context training and prefill.
Useful8/10
Difficulty6/10
Novelty6/10
✗ Failed on benchmark
2026
Replace the usual random or elementwise-positive linear-attention feature map with a rank-one positive-semidefinite feature map derived from query and key vectors. For normalized inputs, the resulting kernel is the squared inner product, which is nonnegative and gives a geometrically structured interference pattern that is better suited to associative recall than an arbitrary low-rank feature map.
Useful8/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace layer-local reconstruction in post-training quantization with a sequential objective that explicitly cancels the error already accumulated by the quantized prefix. For each layer, quantize its weights so that its local residual approximately negates the propagated incoming deviation, preserving the teacher trajectory even when the codebook is binary or 4-bit.
Useful8/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a dense directed attention matrix by a collection of K learned source-to-hub-to-target interactions. Each hub corresponds to a directed biclique, allowing many source tokens to communicate with many target tokens using O(NK) rather than O(N^2) pair interactions. The construction preserves asymmetric information flow and can be initialized from a graph cover of high-attention edges.
Useful8/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Compress each layer's KV tensor with a partial Tucker approximation over token and feature axes, then encode the truncation residual with a rotated uniform quantizer. Select token rank, feature rank, and residual bit-width jointly under a global byte budget, allowing values with flat spectra to receive residual bits while keys may receive more low-rank capacity.
Useful8/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace Euclidean or entrywise Kronecker fitting of a layer curvature matrix with its affine-invariant projection onto G = A tensor B. Use the resulting factors as a compact SPD preconditioner in the optimizer, while solving the projection through logarithmic residual partial traces and Armijo line search.
Useful8/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a deep feed-forward block by the fixed point z=phi(Wz+Vx+b), with the recurrent weight W constrained so that the fixed point is unique for every input. The same condition makes forward fixed-point iteration stable and makes implicit differentiation well-conditioned, allowing depth-independent memory usage while providing a measurable spectral failure boundary.
Useful8/10
Difficulty5/10
Novelty4/10
△ Mechanism confirmed, baseline not beaten
2026
Represent a one-dimensional sharp-feature signal by a small unordered set of complex singularities and residues instead of predicting all grid amplitudes. A transformer diffusion model predicts these tokens, and a differentiable meromorphic decoder evaluates the result directly at arbitrary coordinates, avoiding grid-specific interpolation and preserving discontinuity structure.
Useful8/10
Difficulty6/10
Novelty8/10
✗ Failed on benchmark
2026
Compress each hidden layer by retaining directions that are simultaneously reachable from the observed input distribution and observable at the network output. Unlike PCA or SVD, the retained subspace is weighted by downstream task sensitivity, so high-variance but output-irrelevant directions can be removed while low-variance predictive directions are preserved.
Useful8/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace MSE-calibrated scalar quantization with a conservative left-edge quantizer whose thresholds are denser where activation probability and task utility slope are both high. For a monotone utility function, this should preserve high-impact activation regions better than uniform or MSE-optimal bins at the same number of codes, while retaining an explicit rate-versus-quality design rule.
Useful7/10
Difficulty4/10
Novelty7/10
✗ Mechanism failed
2026
Replace ordinary token merging or graph pooling with a learned block map whose output preserves information about a remote target conditioned on the surrounding coarse representation. The paper's majority-spin counterexample gives a concrete failure mode: two microscopic configurations mapped to the same pooled token can imply different predictions for distant variables.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace dense or single-dilation sparse attention with two sequential sparse attention stages whose offsets form co-prime arithmetic progressions. The first stage mixes tokens separated by multiples of M2, the second by multiples of M1; their composition reaches virtual offsets mM2+nM1, providing many structured long-range interactions from only M1+M2-1 physical offset families. Use causal masking and residual connections so the module can replace a standard transformer attention block without…
Useful7/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Construct deep or recurrent networks whose layer weights are correlated across depth with a prescribed power-law covariance, rather than either fully tying or fully independently sampling layers. The paper predicts two usable design boundaries: \(\gamma=1/2\) for divergence of correlation-induced fourth moments and \(\gamma=1\) for loss of summable-correlation flatness.
Useful7/10
Difficulty6/10
Novelty8/10
△ Mechanism confirmed, baseline not beaten
2026
Replace flat all-pairs attention with attention neighborhoods induced by a compatible tree over tokens, patches, nodes, or retrieved items. Retain exact or approximate attention inside nearby tree subtrees and add a path-monotonicity regularizer so semantically distant endpoints are not more similar than intervening tree neighbors.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace full PSD self-attention with a pivoted Cholesky/Nyström approximation whose landmarks are sampled from the unexplained diagonal mass. Tokens with large residual self-similarity are more likely to become landmarks, so the rank budget is spent on difficult regions rather than uniformly selected tokens.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a fixed-size learned latent for an array-valued complex tensor with a variable-length list of continuous rank-one spectral atoms. An encoder predicts candidate receive direction, transmit direction, residual off-grid offsets, and complex gains; the decoder reconstructs the tensor analytically from the array-response formula, so changing the antenna dimensions does not require changing the decoder weights.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Represent a batch of token or feature directions as columns of a matrix X, and construct a complementary feature basis Y whose columns are annihilated by X under a diagonal gauge. Use Y as a second algebraically complementary channel for attention or token mixing, either replacing redundant feature projections or regularizing them toward an exact nullspace relation.
Useful7/10
Difficulty6/10
Novelty7/10