Unverified
2026
Replace a global attention truncation rule with a per-query local-tail budget. For each query token, retain nearby or high-priority keys until the estimated discarded interaction strength is below a target epsilon; this uses the paper's central distinction between local tail mass and the extensive norm of the discarded operator. The resulting attention pattern can allocate long-range computation only to tokens whose local tail is large.
Useful6/10
Difficulty5/10
Novelty5/10
Unverified
2026
Use the number of tokens participating in an attention operation to adapt the projected query/key dimension according to the sharp Johnson–Lindenstrauss bound. Short attention windows receive a dimension limited by n-1, while longer windows use a logarithmic dimension, avoiding a fixed large head dimension when the current attention set contains too few points to require it.
Useful6/10
Difficulty6/10
Novelty6/10
Unverified
2026
Replace a fixed confidence-threshold early-exit rule with a finite-horizon optimal-stopping policy over the model's evolving posterior confidence. The controller stops when the calibrated expected terminal error is no greater than the cost plus expected value of executing another neural block, permitting time-dependent and nonmonotone stopping regions.
Useful6/10
Difficulty5/10
Novelty5/10
Unverified
2026
Make Laplacian latency attention hardware-friendly by quantizing the exponential decay and row normalization to powers of two. Exponentials and divisions can then be implemented with lookup tables and integer shifts, enabling a low-bit attention kernel for neuromorphic and edge accelerators.
Useful6/10
Difficulty4/10
Novelty7/10
Unverified
2026
Replace the fixed number of Sinkhorn iterations used to normalize an attention kernel with a per-example stopping rule based on the local spectral contraction factor. Estimate the remaining marginal error geometrically and stop early on easy examples while retaining extra iterations on difficult or nearly disconnected examples.
Useful6/10
Difficulty4/10
Novelty5/10
Unverified
2026
Replace a uniformly sampled trajectory sequence by a binary temporal partition whose intervals are split only when the observed trajectory makes an excursion larger than a threshold. Encode one summary token per retained leaf, optionally including duration and endpoint displacement, so smooth trajectory regions receive fewer tokens while rapidly changing regions retain resolution.
Useful6/10
Difficulty4/10
Novelty6/10
Unverified
2026
Replace dense token-to-token attention in selected layers with communication through a small number of multi-token hyperedges. Each hyperedge aggregates its incident token states and broadcasts the resulting message back to those tokens, allowing higher-order interactions while reducing the number of pairwise links. Reconstruct hyperedges periodically from cumulative token displacement so stable tokens retain useful groups while rapidly changing tokens are regrouped.
Useful6/10
Difficulty6/10
Novelty5/10
Unverified
2026
Replace top-k token pruning by greedy maximization of a diversity-aware monotone submodular utility under a spacing or coverage constraint. The selector repeatedly chooses the feasible token with the largest marginal utility, avoiding the redundant-token failure mode of independent score ranking while inheriting a constant-factor approximation guarantee under the stated 2-system abstraction.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace a learned dense token-mixing matrix or residual-state transition with a sparse diffusive mixer whose Laplacian has a deliberately small largest Jordan block. Balance the two chain lengths around the central coupling/core, because the paper proves that this minimizes the worst defective transient among the tridiagonal family. Use a scalar residual step size to move the non-consensus spectrum inside the unit disk while preserving the sparse structure.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Use enumerated weighing matrices as sparse orthogonal channel-mixing operators inside MLPs or residual blocks. Their ternary entries reduce multiplication to signed additions, while exact orthogonality prevents amplification or attenuation of feature norms; a trainable fixed-support version can recover expressivity without giving up computational sparsity.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Use a strictly upper-triangular block operator to represent interactions between ordered layers, experts, or token groups, and compute its inverse exactly with a finite Neumann series. Because nilpotency truncates the series after a known number of blocks, the module avoids an iterative solver while retaining controlled long-range interactions.
Useful6/10
Difficulty5/10
Novelty8/10
Unverified
2026
Replace a recursive product implementation of a rational spectral filter with an additive sum of independently evaluated resolvents. Use the layer on a graph Laplacian, token-similarity operator, or other sparse feature operator to obtain a high-order filter without multiplicative roundoff and gradient amplification; the independent solves can also be batched or distributed across devices.
Useful6/10
Difficulty6/10
Novelty6/10
Unverified
2026
Construct a symmetric token affinity matrix and approximate only its spectrally outlying token-mixing modes using a small set of sampled landmark columns. The resulting low-rank operator replaces an \(O(n^2)\) dense mixer by two skinny matrix multiplications, while the paper's residual guarantee predicts that large-magnitude global interaction modes are preserved.
Useful6/10
Difficulty5/10
Novelty5/10
Unverified
2026
Replace dense attention or strictly local windows with a refreshed sparse communication graph whose edge lengths follow a truncated power law. For sequence positions or image patches at distance $r$, sample edges with probability proportional to $(1+r)^{-(d+\alpha)}$; in the regime $0<\alpha<d$, repeated layers should spread information across the sequence much faster than a local window while using only $O(nk)$ edges per layer. The main test is whether this preserves long-range accuracy at…
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Track the dominant rank-r subspace of the gradient covariance online, then use that basis to construct a low-rank adaptive update or a controlled preconditioner. Unlike offline PCA refreshes, the Oja flow continuously follows changing training geometry while preserving orthonormality, potentially reducing the cost of second-order or Shampoo-like methods.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace direct source-to-target attention connectivity by two sparse incidence relations through a set of latent witness tokens. A source token attends only to a small set of witnesses, and each witness attends only to a small set of target tokens; the composed relation is trained to contain exactly one witness for desired pairs and no witnesses for undesired pairs. This produces a controllable sparse attention pattern whose errors can be measured entrywise against a dense teacher or known mask.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Instrument selected neural-network operators with cheap stochastic perturbations and estimate how much their outputs change under finite-precision perturbations. Use the resulting per-operator score to identify unstable kernels and selectively switch them to FP32, compensated accumulation, or a stable reformulation instead of running the entire model at high precision.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace one-shot softmax expert routing with a small number of synchronous routing rounds in which each token resamples an expert with probability proportional to that expert's current load raised to a power \(\alpha>1\). The resulting positive feedback rapidly creates a dominant routing basin, potentially reducing the number of active experts and communication groups at inference while retaining a controllable exploration phase through the initial round or a token-specific score factor.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace independent binary early-exit or token-pruning decisions with a monotone randomized survival process for each token or expert route. A token can lose survival mass at each layer but cannot become active again; the model is trained with a reflected obstacle-style penalty that activates when the predicted value of continuing computation is below the value of stopping plus the compute cost. Mean-field statistics are computed over currently surviving tokens, making routing less sensitive to…
Useful6/10
Difficulty5/10
Novelty5/10
Unverified
2026
Replace fixed sequence-to-sequence attention with a dynamically maintained tree of connected token groups. Groups can be fused to reduce the number of attention units or split when their representation is heterogeneous, while hypergraph connectivity and nestedness ensure that every intermediate hierarchy remains valid.
Useful6/10
Difficulty6/10
Novelty5/10
Unverified
2026
Replace unconstrained pairwise token grouping with a tree whose edges carry independent merge or cut variables. The connected components of the retained edges define a valid partition at every forward pass, while learned edge gates control the amount of token aggregation. A coarse component-level computation can then replace part of dense attention.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Construct attention groups as hyperedges of a linear r-uniform hypergraph: every pair of tokens is allowed to share at most one group, while each token participates in approximately the same number of groups. Apply local attention inside each group and aggregate the outputs across groups. The construction inherits the paper's sharp capacity bound and prevents both redundant pair interactions and high-degree token hubs.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace a dense channel-mixing matrix by a sparse complex generalised weighing matrix W with exactly w nonzero entries in every row and column, then use U=W divided by square root of w as a norm-preserving mixer. Restricting to k=2 gives a real matrix with entries in {+1,-1}; k=4 supports signed phase rotations. The exact isometry should preserve signal and gradient norms while reducing channel-mixing cost.
Useful6/10
Difficulty6/10
Novelty7/10
Unverified
2026
Assign separate sharpness or temperature parameters to two nonlinear subnetworks and anneal them according to a directional chart instead of driving both to their singular limits at the same rate. The optimizer explicitly tracks the ratio of the two scales and changes the schedule when the local Jacobian approaches a stability or bifurcation boundary. This tests whether the order and relative rate of sharpening, rather than only the final activation shape, controls optimization stability and…
Useful6/10
Difficulty5/10
Novelty8/10