✗ Mechanism failed
2026
Distill a teacher's attention into a student by matching sink mass and the normalized content distribution as separate targets rather than applying one KL divergence to the entire attention row. Use the Aitchison distance on the content composition, which compares relative token allocation and prevents a large common sink probability from overwhelming differences between content tokens.
Useful7/10
Difficulty3/10
Novelty7/10
✗ Failed on benchmark
2026
Replace a wide collection of interchangeable near-zero branches with a module whose output is explicitly a quadratic form in the branch-weight Gram matrix. The module preserves the paper's leading-order behavior while making the relevant collective variable explicit and allowing low-rank parameterizations.
Useful7/10
Difficulty5/10
Novelty7/10
✗ Failed on benchmark
2026
Parameterize an entropic OT cost only in directions that can change the transport plan, removing row-plus-column potential directions that are invisible because of OT gauge invariance. Whiten the remaining feature coordinates using their empirical covariance, producing an OT layer whose identifiable parameters have substantially more uniform sensitivity.
Useful7/10
Difficulty5/10
Novelty7/10
✓✓ Beats tuned baseline
2026
Replace ordinary Wasserstein or arithmetic pooling of distribution-valued features with a barycenter whose individual quantile displacements are Huberized. Small changes between input distributions remain averaged quadratically, while a corrupted token, expert, graph neighborhood, or augmentation cannot move the pooled distribution arbitrarily far. The module is especially cheap for one-dimensional distributions represented by fixed quantile vectors.
Useful7/10
Difficulty4/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace multiplicative query-key attention scores with an affinity based on the l1 distance between first-spike latency vectors. For each query token and key token, small latency differences produce large affinity and distant timings decay exponentially, yielding a locality-sensitive attention pattern naturally compatible with leaky spiking neurons.
Useful7/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Represent a graph, set, or attributed network as a measurable Z-valued kernel and train on lifted representatives while explicitly minimizing over node couplings. The quotient objective is invariant to relabeling by construction, while the lifted loss gives a dense correspondence signal that can stabilize graph attention and relational encoders.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Replace an n-by-n attention or token-mixing matrix with two nonnegative rank-r factors having row-simplex constraints and a shared latent column marginal. The induced matrix is exactly doubly stochastic at every accepted update, while applying it to values uses two thin matrix multiplications and never constructs the dense attention matrix.
Useful7/10
Difficulty6/10
Novelty7/10
✓✓ Beats tuned baseline
2026
Build a kernel aggregation layer whose output is a tangent vector field on the unit sphere and whose surface divergence is identically zero by construction. For each source point, use a matrix kernel obtained by applying a surface-rotated gradient in the query variable to a scalar zonal kernel; this is a differential-form version of the paper's matrix-valued construction. The layer can replace attention or message passing when the target dynamics are incompressible, such as spherical fluid…
Useful7/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Add a per-feature or per-token state that accumulates recent stimulation and decays when stimulation is absent, then use a nonlinear decreasing gain to suppress repeatedly activated features. This creates short-term adaptation without changing the core transformer or recurrent weights: familiar inputs are processed with reduced gain, while novel inputs recover their full response.
Useful7/10
Difficulty4/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Use the relaxed QFT tensor-network topology as a trainable norm-preserving mixer inside a neural block, replacing a dense token-mixing matrix or an expensive global convolution. The network learns data-adapted global interactions while retaining structured O(N log^2 N) application and an exact cheap inverse, making it suitable for image tokens, long sequences, or reversible residual blocks.
Useful7/10
Difficulty6/10
Novelty6/10
✓✓ Beats tuned baseline
2026
Represent each input graph by a short ordered sequence of topological tokens rather than a persistence diagram. For each overlapping interval on a shared filtration grid, compute the number of connected components and independent one-dimensional cycles, concatenate these counts with inexpensive graph summary statistics, and process the resulting sequence with a small Transformer. The representation should retain multiscale structural information while avoiding pairwise persistence-diagram…
Useful7/10
Difficulty6/10
Novelty6/10
✓✓ Beats tuned baseline
2026
Replace unconstrained softmax cross-attention with a many-to-many transport matrix whose row and column masses have explicit upper bounds. Compute the attention cost from both feature similarity and pairwise relational disagreement, so a token is attended to only when its relationships to other tokens are jointly compatible. The inequality constraints provide a principled dustbin-free mechanism for ignoring distractor tokens.
Useful7/10
Difficulty6/10
Novelty5/10
✓✓ Beats tuned baseline
2026
Replace a dense mixing or attention matrix on tokens arranged on a Cartesian grid by a product of learned or fixed one-dimensional concentration operators. The layer applies one axis operator at a time, reducing parameter and compute cost while enforcing that the global operator is a positive contraction with controlled spectral leakage.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Replace or augment dot-product attention with a non-increasing radial kernel of pairwise representation distance. The bandwidth is normalized using an estimated local intrinsic dimension and local neighbor scale, creating an explicit locality-controlled attention operator.
Useful7/10
Difficulty5/10
Novelty6/10
✗ Failed on benchmark
2026
Construct overlapping attention windows from a block design instead of using one dense sequence-by-sequence attention matrix. Every token appears in exactly $r$ windows and every token pair co-occurs in exactly $\lambda$ windows, giving uniform coverage and avoiding the uneven connectivity of arbitrary sparse masks.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Penalize short positive feedback cycles in an iterative neural module by suppressing products of absolute Jacobian blocks around the cycle. This targets the mechanism responsible for exponential temperature sensitivity rather than merely penalizing the total Jacobian norm, allowing strong feed-forward paths while controlling recurrent amplification.
Useful7/10
Difficulty7/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Train a matrix-valued neural layer under an exact or near-exact Stiefel constraint while using an l1 or row-group sparsity penalty. During early training, use manifold proximal-gradient steps to identify a stable nonzero support; once the support stops changing, switch to Newton-CG steps restricted to the smooth intersection of the Stiefel tangent space and the fixed-support subspace. This can reduce the number of optimizer iterations needed to obtain sparse, well-conditioned projections.
Useful7/10
Difficulty6/10
Novelty7/10
✗ Failed on benchmark
2026
Use a low-rank Tucker reconstruction as a structured backbone and quantize only its residual after an orthogonal rotation. The rotation preserves residual energy but redistributes it across coordinates, reducing dynamic-range imbalance and making 2- or 4-bit uniform quantization less damaging than direct quantization of the original KV tensor.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace ordinary low-precision multiply-add accumulation in selected neural-network reductions with a two-word floating-point accumulator updated by the paper's branch-free DW-FMA network. The high word retains the main sum and the low word stores the rounding residual, improving cancellation behavior without the control-flow divergence of conditional compensated summation.
Useful7/10
Difficulty5/10
Novelty6/10
✗ Failed on benchmark
2026
Add a causal memory branch whose lag-response function is represented by a Bernstein polynomial with coefficients constrained to produce a nonnegative, decreasing, convex kernel. The branch aggregates past hidden states using this kernel, giving the model a learnable long-memory profile while preventing oscillatory, negative, or increasing historical influence.
Useful7/10
Difficulty4/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace unconstrained token-mixing logits by a symmetric zero-row-sum response matrix generated from positive conductances on a small auxiliary electrical network. The resulting mixer has conservation and positivity structure, while circular minors have a prescribed sign pattern associated with positive grove measurements. This is especially suitable for graph neural networks and attention variants that need stable global diffusion rather than arbitrary dense affinities.
Useful7/10
Difficulty6/10
Novelty7/10
✗ Failed on benchmark
2026
Replace ordinary pairwise attention similarity by an affinity averaged over transformed keys or values. The resulting attention is invariant to the group action on either input and avoids requiring the network to learn identical attention patterns for every rotated or transformed copy.
Useful7/10
Difficulty5/10
Novelty5/10
✗ Mechanism failed
2026
Replace independent pairwise feature matching across augmented or multimodal views with jointly estimated soft permutation matrices constrained to agree through cycles. The paper's multi-view result suggests that independent copies can cross a correspondence-recovery threshold even when every individual pairwise matching is statistically non-informative. In a neural network, this can provide cleaner token, patch, object, or cell alignment targets and can be used either as a differentiable…
Useful7/10
Difficulty5/10
Novelty6/10
✗ Failed on benchmark
2026
Replace quadratic self-attention over a sequence with a bank of K auxiliary exponentially decaying states whose rates are fitted directly from the empirical autocorrelation of the sequence features. Each mode captures a distinct time scale, so the module can represent short- and long-range dependencies with O(TK) computation and O(K) recurrent memory rather than storing all previous tokens. Constrain decay rates to be positive and use the paper's extended Markovian block structure to obtain a…
Useful7/10
Difficulty5/10
Novelty4/10