✓✓ Beats tuned baseline
2026
Replace the standard unit-step modern Hopfield retrieval update with a relaxed step using theta greater than 1, while restricting theta to the theoretically safe interval (0,2). The relaxed map has the same fixed points as ordinary attention and provably decreases the Hopfield energy, so it can move farther toward an attractor per iteration without changing the retrieval objective.
Useful8/10
Difficulty3/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Allocate different entropy budgets to different KV-cache blocks instead of assigning every token and head the same nominal bitwidth. Use the ECASQ Lagrangian so high-variance or attention-sensitive blocks receive more codepoints, while predictable blocks collapse to fewer symbols and become highly compressible. Preserve unbiasedness per scalar or block so reconstructed keys and values have zero mean quantization error conditional on the original tensor.
Useful8/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace direct logit gradient updates for a simplex-valued neural module with a cascade consisting of a passive LTI filter followed by softmax. The filter can provide useful memory or momentum, but its transfer function is constrained to remain strictly passive, preventing the destabilization mechanism identified for nonpassive higher-order replicator dynamics.
Useful8/10
Difficulty5/10
Novelty7/10
✗ Failed on benchmark
2026
Replace ordinary modality-specific residual fusion with a switched observer whose latent correction depends on the currently available channel. The individual channels are allowed to be insufficient to reconstruct the latent state; stability is enforced over the full switching cycle, so complementary intermittent observations can jointly maintain a stable representation.
Useful8/10
Difficulty6/10
Novelty7/10
✗ Failed on benchmark
2026
Replace a standard softmax-gradient update for a probability vector with a two-stage KL Mirror-Prox update. The predictor evaluates the population-dependent cost at the current distribution, and the corrector evaluates it at the predicted distribution, reducing oscillation when routing or attention costs are coupled across tokens or samples.
Useful8/10
Difficulty5/10
Novelty5/10
✗ Failed on benchmark
2026
Replace consecutive or randomly assigned transformed KV coefficients with groups whose variance-volume is approximately equal. Train one equal-size vector-quantizer codebook per group, so a fixed-width cache does not waste its low-rate budget by forcing high-variance and low-variance coordinates into badly mismatched groups. This is a drop-in quantization-layout change that can be applied to keys, values, or both.
Useful8/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Implement causal linear attention in chunks and combine chunk summaries with an associative scan instead of carrying the recurrent state through all chunks sequentially. This preserves the exact causal computation while reducing inter-chunk dependency depth from the number of chunks to its logarithm, enabling substantially more GPU parallelism for long-context training and prefill.
Useful8/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Compress each layer's KV tensor with a partial Tucker approximation over token and feature axes, then encode the truncation residual with a rotated uniform quantizer. Select token rank, feature rank, and residual bit-width jointly under a global byte budget, allowing values with flat spectra to receive residual bits while keys may receive more low-rank capacity.
Useful8/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace token-by-token KV storage after an SSM or recurrent encoder with an online allocate-on-novelty cache. A new slot is created only when the incoming key is sufficiently dissimilar from every stored key; otherwise the incoming value is merged into its nearest slot, so repeated or redundant content does not grow the cache.
Useful8/10
Difficulty4/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace independent edge-type logits in a relational graph neural network with a mean-field fixed-point router derived from a colored ERGM. Each edge's color distribution is influenced by its own relation bias and by the expected number of rainbow triangles it forms with neighboring edges, allowing the model to learn coordinated multilayer structures.
Useful7/10
Difficulty5/10
Novelty7/10
✓✓ Beats tuned baseline
2026
Replace dense graph self-attention with two parallel branches: exact softmax attention only over graph neighbors and a global linear-attention branch that summarizes all nodes through feature-space statistics. A learned node-wise gate interpolates between the branches, allowing locally structured nodes to use sparse attention while retaining a global-information path.
Useful7/10
Difficulty4/10
Novelty5/10
✗ Mechanism failed
2026
Replace Euclidean momentum for selected neural parameters with a mirror or Bregman update, while using the paper's accelerated Newton direction for the objective step. Entropy geometry is especially suitable for softmax MoE routers, while Euclidean or log-barrier geometries can be used for unconstrained or positive parameters.
Useful7/10
Difficulty6/10
Novelty7/10
✗ Failed on benchmark
2026
Replace an explicit attention step by an implicit Euler step that solves a proximal subproblem involving the Hopfield energy. The new state is evaluated inside the softmax self-consistently, which makes the method less sensitive to large step sizes and can prevent explicit attention from overshooting or tunneling between attraction basins.
Useful7/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace flat all-pairs attention with attention neighborhoods induced by a compatible tree over tokens, patches, nodes, or retrieved items. Retain exact or approximate attention inside nearby tree subtrees and add a path-monotonicity regularizer so semantically distant endpoints are not more similar than intervening tree neighbors.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace full PSD self-attention with a pivoted Cholesky/Nyström approximation whose landmarks are sampled from the unexplained diagonal mass. Tokens with large residual self-similarity are more likely to become landmarks, so the rank budget is spent on difficult regions rather than uniformly selected tokens.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Train a fixed-rank neural weight update Y=USV^T with a projector-splitting Runge–Kutta step instead of independently applying Adam or gradient descent to U, S, and V. The update evolves the full low-rank matrix using the neural gradient but performs QR-based factor updates, avoiding S^{-1} and remaining stable when adapter singular values collapse or cross zero. Use a common-base midpoint construction so every internal stage starts from the same U,V basis and remains rank r.
Useful7/10
Difficulty5/10
Novelty6/10
✗ Mechanism failed
2026
Replace dense cross-attention weights with a balanced transport plan whose nonzero query-key edges are maintained by a multiscale active-set procedure. Solve the coarse token-group problem first, lift its support to the fine token grid, add only edges indicated by local cost or marginal residuals, and warm-start the fine problem from the lifted plan. This should provide a principled sparse attention pattern rather than fixing a global top-k pattern before seeing the transport solution.
Useful7/10
Difficulty7/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace ordinary per-channel or per-token KV quantization with a structured orthogonal transform followed by blockwise 3-bit quantization. Use a normalized Walsh-Hadamard transform and small SO(4) rotations to spread outliers across coordinates, quantize the transformed vectors, and exploit orthogonality to rotate queries and attention outputs so unquantized attention remains mathematically equivalent.
Useful7/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Parameterize a multi-relational graph kernel as a finite stochastic block model and fit it by maximum entropy subject to differentiable motif-density constraints. Use the resulting block kernel as a graph-neural-network message-passing operator or structured prior for edge prediction, reducing an O(n^2 r) relation tensor to O(m^2 r+n) parameters for m latent blocks and r relations.
Useful7/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a portion of quadratic key-value attention or an external episodic table with a per-sample matrix fast memory updated by rank-one delta corrections. The memory directly learns a linear key-to-value map and can be carried across sequence segments, providing cheap online adaptation with constant state size per head.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Penalize short positive feedback cycles in an iterative neural module by suppressing products of absolute Jacobian blocks around the cycle. This targets the mechanism responsible for exponential temperature sensitivity rather than merely penalizing the total Jacobian norm, allowing strong feed-forward paths while controlling recurrent amplification.
Useful7/10
Difficulty7/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Train a matrix-valued neural layer under an exact or near-exact Stiefel constraint while using an l1 or row-group sparsity penalty. During early training, use manifold proximal-gradient steps to identify a stable nonzero support; once the support stops changing, switch to Newton-CG steps restricted to the smooth intersection of the Stiefel tangent space and the fixed-support subspace. This can reduce the number of optimizer iterations needed to obtain sparse, well-conditioned projections.
Useful7/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace dense query-key attention with an adaptive cross approximation constructed from selected query and key pivot tokens. At each rank, choose the pivot pair that removes large estimated residual energy, update the residual by a rank-1 cross correction, and stop when the residual estimate reaches a target tolerance. The resulting factorization computes approximate attention using a small number of landmark interactions while adapting to the actual token distribution.
Useful7/10
Difficulty6/10
Novelty5/10
✗ Failed on benchmark
2026
Use a smoothed Burg entropy as the mirror map in a proximal-gradient optimizer for positive or simplex-valued neural parameters. The optimizer performs a Bregman-proximal step instead of an additive Euclidean update, while the smoothing parameter avoids the singularity of ordinary Burg entropy at zero.
Useful7/10
Difficulty5/10
Novelty5/10