△ Mechanism confirmed, baseline not beaten
2026
Replace dense query-key attention with an adaptive cross approximation constructed from selected query and key pivot tokens. At each rank, choose the pivot pair that removes large estimated residual energy, update the residual by a rank-1 cross correction, and stop when the residual estimate reaches a target tolerance. The resulting factorization computes approximate attention using a small number of landmark interactions while adapting to the actual token distribution.
Useful7/10
Difficulty6/10
Novelty5/10
✗ Mechanism failed
2026
Replace dense self-attention by a sparse mask on a one-dimensional or ordered token geometry, retaining local neighbors and adding long-range edges with probability proportional to distance raised to \(-(1+\sigma)\). Use \(\sigma\approx0.8\text{--}0.85\) as the initial regime because the paper finds that this range supports delocalized, GOE-like connectivity despite sparse bonds. The resulting layer has linear or near-linear attention cost while maintaining long-range paths.
Useful7/10
Difficulty4/10
Novelty6/10
✗ Failed on benchmark
2026
Add a causal memory branch whose lag-response function is represented by a Bernstein polynomial with coefficients constrained to produce a nonnegative, decreasing, convex kernel. The branch aggregates past hidden states using this kernel, giving the model a learnable long-memory profile while preventing oscillatory, negative, or increasing historical influence.
Useful7/10
Difficulty4/10
Novelty7/10
✗ Failed on benchmark
2026
Partition tokens into a rectangular grid and use Latin-square labels to define several sparse attention heads. Each head attends only to tokens sharing a row, column, or Latin label, while orthogonality guarantees that every pair of labels occurs at most once, reducing systematic blind spots and repeated collisions. The resulting masks are deterministic, reusable across examples, and can be generated without learned routing scores.
Useful7/10
Difficulty5/10
Novelty7/10
✗ Failed on benchmark
2026
Construct a weight-tied transformer loop in which the recurrent state receives a bounded diagonal carry plus a learned block increment, rather than applying a residual identity inside the learned increment. Parameterize the carry so every channel is strictly below one, allowing many recurrent iterations without the state explosion observed with an unconstrained carry.
Useful7/10
Difficulty4/10
Novelty5/10
✓✓ Beats tuned baseline
2026
Route tokens to experts using Laguerre cells defined by the minimum control energy needed to move a token embedding to each expert prototype, rather than by Euclidean distance or an unconstrained learned router logit. Per-expert dual weights deform the cells so that minibatch routing follows prescribed expert capacities, giving a geometrically interpretable alternative to auxiliary load-balancing losses.
Useful7/10
Difficulty6/10
Novelty6/10
✗ Failed on benchmark
2026
Replace ordinary pairwise attention similarity by an affinity averaged over transformed keys or values. The resulting attention is invariant to the group action on either input and avoids requiring the network to learn identical attention patterns for every rotated or transformed copy.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Convert a sequential modular network into parallel execution layers by placing mutually commuting operators in the same layer. The resulting circuit preserves all noncommuting precedence constraints while exposing safe concurrency and fusion opportunities for inference or training.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a conventional two-layer channel mixer in a reversible architecture with the tropicalization of two cluster mutations. For every pair of channels, the block applies sign-dependent integer shears and reflections, giving a cheap piecewise-linear transformation that is exactly invertible and requires no stored activations during backpropagation. Continuous trainable affine scale and mixing parameters can be placed around the fixed tropical core.
Useful7/10
Difficulty4/10
Novelty8/10
△ Mechanism confirmed, baseline not beaten
2026
Build a multi-expert or multi-task layer whose feature channels are divided into a globally shared subspace and expert-private subspaces. Matrix-weighted message passing couples experts only through selected feature directions, while the nullspace preserves specialization; the graph-cut condition provides a concrete test that the shared channels can propagate across all experts rather than becoming disconnected islands.
Useful7/10
Difficulty6/10
Novelty7/10
✗ Failed on benchmark
2026
Replace a weight-tied residual or neural-ODE stepper with an explicit Runge–Kutta method satisfying the reused-last-stage conditions. The final derivative is evaluated at the exact endpoint and becomes the first derivative of the next step, saving one expensive neural-vector-field call per step while preserving the designed integration order.
Useful7/10
Difficulty5/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Use the paper's finite-width O(n^{-1/2}) Gaussian-process approximation bound as a width-budgeting rule rather than choosing every hidden dimension uniformly. Estimate an architecture-specific constant for each layer or attention contraction, then allocate width according to the smallest dimension satisfying its allowed distributional error. This should produce narrower models at comparable GP-like behavior, or permit the same parameter budget to be concentrated in the layers where finite-width…
Useful7/10
Difficulty5/10
Novelty7/10
✗ Failed on benchmark
2026
Replace dense coarse-to-fine cross-attention at multiresolution interfaces with a sparse, nonnegative overlap operator whose weighted feature average is exactly conserved between the two resolutions. Use this operator as a low-order path and blend it with an unrestricted neural cross-attention path through a convex limiter that keeps features inside a box or simplex domain. The construction is especially suitable for adaptive token grids, hierarchical graph neural networks, neural operators…
Useful7/10
Difficulty5/10
Novelty8/10
✗ Mechanism failed
2026
Replace a learned binary MoE gate with a hyperplane whose two sides contain prescribed proportions of several token populations simultaneously. In a low-dimensional routing projection, solve the cap-volume equations directly, producing deterministic per-population load control without an auxiliary load-balancing loss. Recursively applying the construction yields a balanced binary expert tree.
Useful7/10
Difficulty6/10
Novelty7/10
✗ Mechanism failed
2026
Add a deterministic complex-valued state-space bank whose mode detunings become progressively smaller with mode index, Delta_n=c n^{-p}, while input couplings decay as B_n=b n^{-kappa}. For slowly varying or constant forcing, the summed state follows the paper's subresonant response and grows like t^{1-alpha}, providing controllable power-law memory with only O(N) recurrent state updates. This should improve long-context retention compared with a same-size unconstrained RNN or uniformly spaced…
Useful7/10
Difficulty5/10
Novelty7/10
✗ Mechanism failed
2026
Construct a sparse attention layer by sampling backward token histories as a continuous-time branching process rather than allowing every query to attend to every key. Each active ancestor either dies or branches into a bounded number of candidate ancestors, with branching probability controlled by a small parameter. The branch-out penalty predicts exponentially small probability of long, highly branching histories, providing a direct knob for receptive-field size and attention FLOPs.
Useful7/10
Difficulty5/10
Novelty7/10
Audited (legacy)
2026
Replace uniform embedding dimensions with a globally budgeted allocation based on the estimated spectral complexity of each categorical feature. Tables whose category representations have large leading singular-value energy receive more dimensions, while high-cardinality tables are penalized because each extra dimension consumes more parameters.
Useful7/10
Difficulty4/10
Novelty6/10
Unverified
2026
Replace an unconstrained attention or MoE routing matrix with a row-stochastic matrix that preserves a learned Gibbs-like distribution over groups. Anneal its temperature so that the router converges to a block-upper-triangular operator: mixing remains flexible within equal-energy groups, while cross-group traffic becomes directional rather than oscillating or collapsing. This creates a hierarchy-aware inductive bias and can reduce the number of active expert or attention connections.
Useful6/10
Difficulty6/10
Novelty7/10
Unverified
2026
Replace dense token attention with K asynchronous local nonlinear consensus updates on a sparse token graph. Each selected token is moved to the unique incident p-mean of its neighbors, producing a tunable message-passing operator that is averaging for p=2 and nonlinear for other p.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace dense long-range token interactions with sparse message passing on a balanced hierarchical tree. Boundary token features are extended to internal tree nodes by weighted harmonic relaxation, and the resulting weighted normal derivative is used as a learned multiscale interaction or regularization signal.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace heuristic softmax load balancing with a convex entropic projection that computes expert assignment fractions satisfying explicit linear capacity and load constraints. The projection is bounded componentwise and keeps poorly identified routing decisions away from extreme 0/1 allocations, reducing expert collapse and overload under noisy or shifted batches.
Useful6/10
Difficulty6/10
Novelty6/10
Unverified
2026
Construct a sparse directed attention mask over ordered tokens by treating token pairs as edges labeled by time or position. Apply a recursive three-hop spanner construction so that every dense temporal route has a sparse route using at most three times as many hops. Run attention only on the selected edges and use three sparse message-passing layers, or a small learned hop-composition block, to approximate dense causal attention.
Useful6/10
Difficulty6/10
Novelty6/10
Unverified
2026
Insert a projection-space mixer that combines several fixed or learned directions using reciprocal correlations with the current feature, then normalize the result. The exact construction has a universal beta law for its squared input-output cosine, so it can create controlled angular diversity while remaining deterministic and independent of the chosen direction dictionary.
Useful6/10
Difficulty5/10
Novelty8/10
Unverified
2026
Replace unconstrained token-to-expert routing with a nonnegative mixture of a small set of feasible routing or communication patterns. Enforce resource limits using the paper's one-sided positive-contribution bound, which gives a conservative certificate without enumerating all joint token activation scenarios. Expand the pattern set only when a separation procedure finds a routing pattern that improves the router objective while adding useful capacity information.
Useful6/10
Difficulty6/10
Novelty7/10