✗ Failed on benchmark
2026
Augment a neural router with the age of its current expert or latent regime and use an age-dependent hazard to determine when switching is likely. Unlike ordinary token-wise softmax routing, the router can learn non-geometric residence times, suppressing unstable expert oscillations while still allowing rapid transitions when the current regime becomes inappropriate.
Useful6/10
Difficulty5/10
Novelty7/10
✓✓ Beats tuned baseline
2026
Replace a recurrent token-mixing operation with a finite-state carrier scan over binary or quantized token features. The local transition table is constrained to conserve a scalar token weight and to be bijective, making the mixer reversible, constant-memory, and less prone to activation drift than a generic recurrent layer.
Useful6/10
Difficulty5/10
Novelty8/10
✗ Mechanism failed
2026
Replace the first dense layer on q-ary categorical features by a Fourier interaction layer containing only monomials whose coordinate support is at most s. Use a Bohnenblust–Hille-inspired quasi-norm on coefficients, separately for each interaction order, to prevent a small number of high-order interactions from dominating the output. The resulting model has an explicit interaction-order knob and can be tested against a dense MLP at matched parameter count.
Useful6/10
Difficulty5/10
Novelty7/10
✗ Mechanism failed
2026
Replace an unconstrained scalar activation or nonnegative gate with a finite positive mixture of rational Bernstein basis functions. The learned function is monotone and concave on the nonnegative half-line, while its derivatives have controlled alternating signs; this can prevent pathological feature amplification and gives an interpretable shape prior. Use the paper's sharp exponent restriction τ≤1/2 rather than treating the power as an arbitrary hyperparameter.
Useful6/10
Difficulty4/10
Novelty7/10
✓ Mechanism works
2026
Replace a flat MoE or graph-pooling assignment with recursive partitions selected by interaction evidence after removing each item’s expected degree effect. Tokens, nodes, or examples that are frequently active for purely popularity-related reasons should not automatically form an expert or cluster. Recursion stops when a candidate split has nonpositive degree-corrected evidence, producing an adaptive hierarchy rather than a fixed number of equally sized groups.
Useful6/10
Difficulty6/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace ordinary one-token-to-one-expert or one-token-to-one-attention routing with a local latent subset router: a pooled observation can be explained by a compatible subset of tokens. Pairwise compatibility scores assign probability to subsets, and each token receives the marginal probability that it belongs to the selected subset. This should help when tokens represent overlapping objects, occluded entities, or multiple features that should be processed jointly.
Useful6/10
Difficulty5/10
Novelty6/10
✗ Mechanism failed
2026
Replace a free-form order-dependent gate with a positive mixture of Mellin powers $(1+s)^{-a}$. This gives a small, interpretable module whose response across the order variable is automatically generated by a positive measure and therefore inherits complete monotonicity, log-convexity, and Hankel-moment structure.
Useful6/10
Difficulty4/10
Novelty8/10
✗ Mechanism failed
2026
Replace a flat stochastic MoE router by a two-level continuous-time routing model: experts within a group communicate rapidly, while transitions between groups occur slowly. Use the effective class-level stationary distribution as a soft load-balancing prior, reducing routing oscillation while preserving expert specialization.
Useful6/10
Difficulty6/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Parameterize a relative-position or lag-decay function as a finite positive mixture of exponentials instead of learning arbitrary attention bias values. The resulting kernel is completely monotone on positive distances, so it is nonnegative, decreasing, and has alternating derivative signs; the mixture provides several learned memory scales without allowing oscillatory or unstable long-range biases.
Useful6/10
Difficulty4/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace an independently generated discrete latent sequence or redundancy stream with a symmetric two-state Markov source whose transition probability is tuned or learned. Train the downstream transformer to reconstruct the semantic target after random insertions, while using the paper's insertion-capacity expression to select the latent rate and redundancy budget. The representation should preserve information under timing drift, repeated tokens, and inserted distractors better than iid token…
Useful6/10
Difficulty5/10
Novelty7/10
✗ Mechanism failed
2026
Replace an unconstrained high-order polynomial interaction module with features generated by Gaussian matrix contractions and their exact Wick expansion. The resulting interactions are sums of products of power-sum invariants, with coefficients fixed by perfect-matching counts, providing a low-parameter inductive bias for permutation- or orthogonal-structured data.
Useful6/10
Difficulty5/10
Novelty7/10
Audited (legacy)
2026
Use the attention probability distribution over an ordered context to choose contiguous token groups whose pooled attention masses have entropy as close as possible to a prescribed upper budget R. Replace the corresponding key/value vectors by one weighted representative per group, preserving token order and reducing the KV-cache length from n to m. Unlike unconstrained token merging, the entropy constraint gives a direct control knob over how concentrated or diffuse the retained attention…
Useful6/10
Difficulty5/10
Novelty7/10
✓✓ Beats tuned baseline
2026
Estimate the temporal spectrum of each sequence channel using a locally private procedure, then apply a regularized inverse-square-root spectral filter before the sequence enters attention or an SSM. The filter removes predictable low-frequency or narrow-band redundancy while avoiding unstable amplification at frequencies where the private estimate is small.
Useful6/10
Difficulty5/10
Novelty7/10
✗ Mechanism failed
2026
Represent candidate two-dimensional attention windows as dyadic rectangles and penalize local regions where many deeply embedded windows overlap. Use complementary horizontal and vertical depth exponents rather than independently penalizing one coordinate. The resulting router should reduce pathological concentration of sparse attention computation while preserving access to multiscale context.
Useful6/10
Difficulty6/10
Novelty7/10
✗ Failed on benchmark
2026
Replace the usual hand-designed expert-load penalty with a heterogeneous survival penalty derived from a susceptibility distribution. Each expert receives an availability factor q_e=G(A_e), where A_e is its cumulative recent routing pressure and G_e is a learned or fixed mixture of exponentials; highly used experts are suppressed smoothly, while heterogeneous experts can have different resistance to pressure. The mixture produces adaptive curvature and long-tailed penalties that may reduce…
Useful6/10
Difficulty4/10
Novelty6/10
Unverified
2026
Replace a dense weak pairwise interaction over tokens or graph nodes by an analytically resummed Gaussian quadratic term plus explicitly evaluated corrections containing only subsets of size at most R. Choose R from the target approximation error and interaction scale instead of fixing the interaction order heuristically. This is intended for dense global attention, graph interactions, or latent-variable energy models with individually small couplings.
Useful5/10
Difficulty7/10
Novelty8/10
Unverified
2026
Replace independently random unit-normalized prototypes in a spherical classifier, vector-quantizer, or prototype contrastive head with prototypes selected to cover the sphere evenly. The method directly targets the paper's finding that weak-signal spherical K-means preserves initialization-induced Voronoi structure, reducing redundant prototypes and making early assignments less dependent on random seed.
Useful5/10
Difficulty4/10
Novelty5/10
Unverified
2026
Partition activations into dyadic magnitude bands and allocate sparse connectivity separately to heavy and diffuse coordinates. Protect high-magnitude coordinates with more reliable connections while using randomized flat connectivity for the many small coordinates, keeping the total number of nonzeros fixed.
Useful5/10
Difficulty6/10
Novelty7/10
Unverified
2026
Construct positional or relative-position features as a nonnegative mixture of lattice cosine functions instead of independently signed sinusoidal features. The resulting bias is the Fourier transform of a positive discrete measure with explicitly bounded spectral support, while the mesh and degree can be initialized in the paper's dense-but-controlled frequency regime.
Useful5/10
Difficulty4/10
Novelty7/10
Unverified
2026
Replace a deterministic mixture-of-experts residual block with K population-indexed stochastic expert states coupled through a graphon matrix. The layer uses a shared drift and expert-dependent diffusion, while an empirical convex-order penalty makes later representations more dispersed than a reference representation without permitting a mean shift.
Useful5/10
Difficulty5/10
Novelty7/10
Unverified
2026
Replace dense attention on tree-structured inputs with stochastic attention neighborhoods formed by metric balls of sampled radii. Use the paper's exact trimming rule to ensure that every sampled cover remains valid while its total radius budget is bounded, then average predictions over several independent covers during training. This creates sparse, globally covering attention masks with an explicit locality-versus-coverage control.
Useful5/10
Difficulty6/10
Novelty8/10
Unverified
2026
Build a binary hierarchy over tokens by recursively splitting each active block with a beta-splitting rule, then perform dense attention only inside small leaf blocks and communicate between leaves through learned summaries at internal nodes. The beta parameter controls how balanced the partition is, while the paper's maximum-depth asymptotic supplies a principled depth budget and a way to detect pathological trees.
Useful5/10
Difficulty6/10
Novelty5/10
Unverified
2026
Replace dense token-to-token attention on a 2D token grid with local attention plus sparse horizontal and vertical communication axes. Tokens at intersections of selected axes receive extra cross-axis attention edges, creating a reinforced sparse graph that can transmit information across large blocks while using far fewer edges than dense attention. The mask should use light-tailed, approximately geometric spacing in both directions rather than heavy-tailed spacing in one direction.
Useful5/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace a polynomial layer's single-replica output statistics with a finite fingerprint computed from several correlated Gaussian replicas. Train the fingerprint to be invariant under orthogonal reparameterizations while remaining discriminative between genuinely different polynomial maps, preventing models from collapsing distinct tensor functions that have identical marginal output laws. This is a practical symmetry-aware regularizer or auxiliary embedding for tensorized MLPs and polynomial…
Useful5/10
Difficulty5/10
Novelty8/10