✓✓ Beats tuned baseline
2026
Use an approximate decision diagram to select a structured subset of neurons, channels, attention heads, or attention edges when their quadratic interactions are sparse or inverse-sparse. Merge states that agree on a local interaction boundary and accept a tunable epsilon loss in the pruning objective, obtaining a representation whose size is linear in model width for fixed accuracy tolerance.
Useful6/10
Difficulty6/10
Novelty7/10
✗ Mechanism failed
2026
Represent candidate two-dimensional attention windows as dyadic rectangles and penalize local regions where many deeply embedded windows overlap. Use complementary horizontal and vertical depth exponents rather than independently penalizing one coordinate. The resulting router should reduce pathological concentration of sparse attention computation while preserving access to multiscale context.
Useful6/10
Difficulty6/10
Novelty7/10
✗ Failed on benchmark
2026
Replace the usual hand-designed expert-load penalty with a heterogeneous survival penalty derived from a susceptibility distribution. Each expert receives an availability factor q_e=G(A_e), where A_e is its cumulative recent routing pressure and G_e is a learned or fixed mixture of exponentials; highly used experts are suppressed smoothly, while heterogeneous experts can have different resistance to pressure. The mixture produces adaptive curvature and long-tailed penalties that may reduce…
Useful6/10
Difficulty4/10
Novelty6/10
✗ Failed on benchmark
2025
Replace a conventional scalar activation by a geometrically indexed family of affine pieces whose slope changes with the logarithmic magnitude of the input. The same two endpoint parameters are reused across all scales, giving a compact, explicitly scale-aware activation that can represent different responses for exponentially separated activation magnitudes.
Useful6/10
Difficulty4/10
Novelty7/10
Unverified
2026
Replace an unconstrained Cartesian-product router over heterogeneous branches with a router whose joint expert or state assignments obey a finite-group conservation rule. Branch i emits a distribution over labels in its own subgroup H_i of a common finite abelian group G; only tuples whose group sum is zero are retained. This gives an exact, differentiable structural prior for modular arithmetic, multi-relational graphs, multi-view fusion, or any setting where latent labels compose by a…
Useful5/10
Difficulty5/10
Novelty7/10
Unverified
2026
Replace a dense or ad hoc sparse attention pattern with a circulant mask generated by the paper's carry word c(m,k). Every query attends to exactly m of k relative positions, and the selected positions are prefix-balanced, avoiding the large gaps and collisions produced by random sparsification. Use several Farey-related slopes across heads or layers to combine local, medium-range, and long-range coverage.
Useful5/10
Difficulty5/10
Novelty7/10
Unverified
2026
Replace a static top-k MoE capacity rule with a router whose expert allocation evolves through a finite-domain coverage process. Experts with larger current occupancy can either receive more future capacity, intentionally amplifying specialization, or receive less capacity by reversing the size dependence, allowing a controlled test of the paper's asymmetry-amplification mechanism.
Useful5/10
Difficulty6/10
Novelty6/10
Unverified
2026
Use the adjacency matrix of a vertex-transitive strongly regular graph as a fixed sparse attention or token-mixing mask. Every vertex has the same degree, and every pair of vertices has exactly one of two common-neighbor counts, giving predictable two-hop coverage and avoiding the degree and connectivity irregularities of random sparsification.
Useful5/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace random or learned routing hashes for tokens arranged on a d by d grid with a fixed family of mutually orthogonal anti-Latin squares. Each channel assigns exactly d of the d squared tokens to every bucket, while any two channels jointly distinguish every grid position. The resulting router has deterministic load balance and multi-view positional diversity.
Useful5/10
Difficulty5/10
Novelty7/10
Unverified
2026
Replace or augment standard sinusoidal or RoPE position features with bracket-quadratic phases $e(-\theta n\lfloor\beta n\rfloor)$ generated by a Heisenberg nilmanifold orbit. Multiple irrational coefficients and output frequencies produce a cheap deterministic encoding whose empirical cross-position correlations should exhibit cancellation instead of the periodic aliasing of rational or finite-frequency encodings.
Useful5/10
Difficulty3/10
Novelty7/10
Unverified
2026
Replace a dense token- or channel-mixing matrix with a product of local braid generators acting on adjacent coordinates. Each generator is an exactly invertible 2-by-2 transformation, while the braid and far-commutativity identities give multiple equivalent factorizations of the same global operator. This creates a sparse, reversible mixer with O(kn) cost for a braid word of length k, rather than O(n^2) cost for a dense matrix.
Useful5/10
Difficulty4/10
Novelty7/10
Unverified
2026
Design sparse attention masks using a graph discrepancy criterion rather than selecting only local or nearest-neighbor edges. A mask with approximately uniform edge counts between every pair of token subsets spreads information globally, while the rigidity consequence provides a principled way to preserve enough independent pairwise constraints for latent geometric features.
Useful5/10
Difficulty4/10
Novelty6/10
Unverified
2026
Replace an unconstrained three-token interaction block by three distinct pair maps constructed from anticommuting channel generators. For every token triple, enforce equality of the two composition paths A12 B13 C23 and C23 B13 A12, while retaining different parameters for the three edges. This creates a globally consistent three-way interaction without collapsing to a single shared pair operator.
Useful5/10
Difficulty6/10
Novelty9/10
Unverified
2026
Replace dense attention on tree-structured inputs with stochastic attention neighborhoods formed by metric balls of sampled radii. Use the paper's exact trimming rule to ensure that every sampled cover remains valid while its total radius budget is bounded, then average predictions over several independent covers during training. This creates sparse, globally covering attention masks with an explicit locality-versus-coverage control.
Useful5/10
Difficulty6/10
Novelty8/10
Unverified
2026
Build a binary hierarchy over tokens by recursively splitting each active block with a beta-splitting rule, then perform dense attention only inside small leaf blocks and communicate between leaves through learned summaries at internal nodes. The beta parameter controls how balanced the partition is, while the paper's maximum-depth asymptotic supplies a principled depth budget and a way to detect pathological trees.
Useful5/10
Difficulty6/10
Novelty5/10
Unverified
2026
Use the paper's eventual path-length bounds to constrain an order-invariant routing graph to a constant-hop communication budget. A learned sparse attention or graph-neural-network layer can explicitly route information through at most three admissible hops, while a more conservative auxiliary route permits at most five minimal-path hops, preventing increasingly long and unstable dependency chains as sequence length grows.
Useful5/10
Difficulty7/10
Novelty8/10
Unverified
2026
Replace dense token-to-token attention on a 2D token grid with local attention plus sparse horizontal and vertical communication axes. Tokens at intersections of selected axes receive extra cross-axis attention edges, creating a reinforced sparse graph that can transmit information across large blocks while using far fewer edges than dense attention. The mask should use light-tailed, approximately geometric spacing in both directions rather than heavy-tailed spacing in one direction.
Useful5/10
Difficulty5/10
Novelty6/10
Unverified
2026
Constrain a sparse attention graph to split into a k-degenerate backbone and a residual graph with maximum degree at most k-1. Orient the backbone according to a degeneracy order so that each token receives or emits at most k structured interactions in the relevant direction, while the residual edges form a bounded-degree correction layer. This replaces arbitrary sparse attention with a topology that is easier to schedule and whose worst-case edge and local-degree costs are explicit.
Useful5/10
Difficulty6/10
Novelty8/10
Unverified
2026
Construct a sparse token-mixing architecture by interleaving learned per-token transformations with fixed perfect-shuffle and cyclic-pile permutations. For n not a power of k, the generated permutation group is 2-transitive, so sufficiently rich sequences of generator words can expose every ordered token pair without constructing a dense N by N attention matrix.
Useful5/10
Difficulty4/10
Novelty6/10
Unverified
2026
Build a differentiable assignment layer whose rows represent tokens and whose columns represent experts, memory slots, or attention slots. Each row has unit probability mass, but no column receives positive mass from two rows; maintaining at least one vacant column makes assignments continuously deformable through elementary vacancy moves instead of abrupt softmax switches.
Useful5/10
Difficulty6/10
Novelty5/10
Unverified
2026
Replace a dense attention pattern by the exact intersection of a fixed or cheaply computed base graph H and a learned shared-label relation. Two tokens can exchange information only when they are adjacent in H and share at least one of d labels, producing a controllable structured sparsity pattern. The label count d becomes an explicit capacity and compute knob: increasing d enlarges the relation vocabulary without requiring a dense pairwise mask.
Useful5/10
Difficulty6/10
Novelty5/10
Unverified
2026
Replace part of an attention matrix with a mixture of fuzzy permutation matrices induced by short permutations. Each basis element represents an order-preserving k-token matching smeared over all embeddings into the sequence, while a balancing constraint makes the aggregate attention receive uniform global coverage. Retain a standard low-rank or local-attention residual so the structured branch does not prevent arbitrary content-dependent interactions.
Useful5/10
Difficulty5/10
Novelty7/10
Unverified
2026
Add a small number of latent region-offset variables to a graph or token-mixing layer, interpreting selected edges as low-permeability barriers that suppress cross-region information flow. Eliminate the latent variables analytically, yielding a visible-node update with a structured low-rank correction rather than adding persistent hidden node states. The module is intended to preserve within-cluster propagation while preventing oversmoothing or contamination across learned boundaries.
Useful5/10
Difficulty6/10
Novelty6/10
Unverified
2026
Replace or augment an exponential state-space memory branch with a causal convolution whose lag-j weight is exp(-lambda j) ell(j)/j. The 1/j boundary provides broad logarithmic memory, while lambda supplies an explicit finite memory scale and prevents uncontrolled accumulation from an untempered long-memory kernel.
Useful5/10
Difficulty5/10
Novelty5/10