△ Mechanism confirmed, baseline not beaten
2026
Replace selected dense neural-network operators by low-rank factors whose rank is selected by a randomized residual test at a user-specified tolerance. Construct candidate bases in large blocks for efficient matrix operations, then prune the block to the smallest rank that passes the residual criterion instead of treating the block size as the final rank.
Useful7/10
Difficulty5/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Represent a high-order feature tensor as a tensor train and replace a dense global feature transform by a truncated polynomial in a learned nearest-neighbor operator. The block computes a short Krylov expansion, p_m(A)x = sum from k=0 to m of c_k A^k x, compressing back to a fixed TT rank after each operator application; locality is intended to prevent rank growth from scaling with the total number of tensor sites.
Useful7/10
Difficulty6/10
Novelty5/10
△ Mechanism confirmed, baseline not beaten
2026
Use the paper's finite-width O(n^{-1/2}) Gaussian-process approximation bound as a width-budgeting rule rather than choosing every hidden dimension uniformly. Estimate an architecture-specific constant for each layer or attention contraction, then allocate width according to the smallest dimension satisfying its allowed distributional error. This should produce narrower models at comparable GP-like behavior, or permit the same parameter budget to be concentrated in the layers where finite-width…
Useful7/10
Difficulty5/10
Novelty7/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a dense multiresolution voxel or hash-grid encoder with an omnitree-like anisotropic feature partition. Each cell stores a vector-valued scaling feature and its children are introduced only when local Haar detail energy is large; coarsening replaces children by their mean, so compression does not introduce an arbitrary offset. Splitting can be restricted to the coordinate whose one-dimensional detail coefficient is largest, allowing thin structures to receive resolution only in the…
Useful7/10
Difficulty5/10
Novelty7/10
✗ Mechanism failed
2026
Replace the dense decoder of an overcomplete activation autoencoder with a fixed left-d-regular expander mask and learn only the nonzero decoder values. Tie the encoder to the transpose of this sparse decoder, preserving the activation dimension m, latent width n, and TopK sparsity k while reducing learned dictionary parameters from mn to dn.
Useful7/10
Difficulty5/10
Novelty7/10
Audited (legacy)
2026
Replace uniform embedding dimensions with a globally budgeted allocation based on the estimated spectral complexity of each categorical feature. Tables whose category representations have large leading singular-value energy receive more dimensions, while high-cardinality tables are penalized because each extra dimension consumes more parameters.
Useful7/10
Difficulty4/10
Novelty6/10
✗ Mechanism failed
2026
Replace an opaque MLP vector field with a stack of trainable symbolic primitives that can express linear terms, monomials, products, and related analytic operations. Apply an L1 penalty and prune small primitive coefficients after rollout training, yielding a compact dynamics module that is cheaper to evaluate and easier to inspect.
Useful7/10
Difficulty6/10
Novelty6/10
Unverified
2026
Replace a dense token-mixing, MLP, or low-rank-adapter projection with a Bernoulli-signed sparse matrix normalized by the paper's predicted operator scale. Estimate the active representation dimension and use the effective scale to detect regimes in which extreme sparsity is likely to cause unstable amplification or dead rows.
Useful6/10
Difficulty4/10
Novelty5/10
Unverified
2026
Replace a dense directional feature map with a finite stable-zonotope layer: a small set of large random vectors creates a hyperplane arrangement over the input sphere, while a cheaper small-jump residual gives continuous features inside each cell. The layer exposes an explicit compute-versus-resolution knob through the jump cutoff and can be used either as an embedding before an MLP or as a lightweight router for mixture-of-experts blocks.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Replace a dense unconstrained channel-mixing matrix with a differentiable product of exponentials of a few skew-symmetric generators and their iterated commutators. The resulting layer is exactly orthogonal, preserves feature norms, and can express rotations in directions not explicitly stored as independent parameters. This is especially suitable for residual MLP blocks, recurrent state transitions, and networks processing rotation- or pose-valued features.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Construct a sparse fixed orthogonal mixer by repeatedly applying pi/4 rotations to randomly matched pairs of feature coordinates. Place this mixer before top-k feature pruning, sparse projection, or activation quantization so that information is spread across coordinates without using a dense random matrix.
Useful6/10
Difficulty4/10
Novelty6/10
Unverified
2026
Build each nonlinear correction in an inverse neural operator from explicit bilinear products of learned operator features, following the inverse Born expansion instead of using an unconstrained pointwise MLP. Use a square activation to implement multiplication exactly, and truncate the interaction order so the model has a controllable polynomial structure.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Represent hierarchical or tree-structured hidden states on words over d symbols and replace a dense mixing layer by a noncommutative Toeplitz operator composed of shared word shifts. Coefficients are reused at every tree location, so the parameter count depends on maximum interaction depth rather than the number of nodes; an optional spectral penalty controls the amplification profile of finite-depth truncations.
Useful6/10
Difficulty5/10
Novelty7/10
Unverified
2026
Replace a dense multiresolution grid encoding for a coordinate MLP with a hierarchical sparse tensor-product encoding whose active cells are selected by local hierarchical surpluses. Combine anisotropic component grids with alternating binomial weights, then refine only regions whose encoded or prediction residual is large. This should preserve fine detail around localized structures while avoiding the exponential parameter count of a full grid.
Useful6/10
Difficulty5/10
Novelty5/10
Unverified
2026
Represent a 3D neural field using high-order polynomial coefficients attached to an adaptively refined tetrahedral mesh, with a small MLP predicting residual corrections from local coordinates. Refine only tetrahedra whose prediction, rendering, or PDE residual is large, and use globally ordered vertices so neighboring tetrahedra share identical face and edge coefficients without hanging-node constraint solves.
Useful6/10
Difficulty6/10
Novelty7/10
Unverified
2026
Separate low-support monomials, which involve only a few distinct input coordinates, from high-support monomials in a high-degree symmetric interaction layer. Compute the low-support orbit features exactly and prune, sample, or factorize the high-support tail, using the paper's cutoff scale as the initial sparsity rule.
Useful6/10
Difficulty5/10
Novelty8/10
Unverified
2026
Add a neural feature, adapter, or expert block only through the component of its outputs that is orthogonal to the span of all previously installed blocks. Quotient coefficient directions that produce nearly identical outputs with an SVD or pseudoinverse, so the new block contributes intrinsic representational dimensions instead of duplicating old features. The expected benefit is a smaller effective architecture and better-conditioned block expansion.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Replace isotropic variance control in a bottleneck or router with a spectral polarization penalty that drives each latent direction toward either variance 0 or variance 1. The intended result is an automatically selected active subspace: inactive coordinates can be pruned or quantized aggressively, while active coordinates retain information instead of being uniformly attenuated.
Useful6/10
Difficulty4/10
Novelty7/10
Unverified
2026
Replace a learned dense bilinear map with a structured family of signed orthogonal matrices. Given feature vectors y,z in R^n, produce r interaction features h_a = y^T H_a z / sqrt(n), where the H_a form a Hadamard/Clifford-like family; the resulting bilinear map has operator norm at most one when r is within the Hurwitz–Radon limit. Learn only channel projections and optional scalar gates around this fixed mixer.
Useful6/10
Difficulty6/10
Novelty7/10
Unverified
2026
Replace globally backpropagated hidden-layer losses with independent recurrent layers that receive bottom-up, top-down, and lateral inputs. Train each layer to assign low activity to correctly paired input-label examples and high activity to mismatched examples, then classify by selecting the label with the lowest accumulated surprise.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Use enumerated weighing matrices as sparse orthogonal channel-mixing operators inside MLPs or residual blocks. Their ternary entries reduce multiplication to signed additions, while exact orthogonality prevents amplification or attenuation of feature norms; a trainable fixed-support version can recover expressivity without giving up computational sparsity.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Represent a neural-network weight tensor by rank-one terms whose mode factors are selected from shared orthonormal bases, and impose the same basis alignment across tensor flattenings. During or after training, retain the largest coefficients to obtain a structured truncation analogous to truncated SVD. This should produce better-conditioned tensorized layers than unconstrained CP factors while preserving a directly controllable accuracy/compute tradeoff.
Useful6/10
Difficulty5/10
Novelty5/10
Unverified
2026
Replace heuristic magnitude pruning in a layerwise convex reconstruction problem with safe screening based on a perspective relaxation of the cardinality constraint. A channel is removed only when a lower bound for every solution containing that channel exceeds the loss of a feasible incumbent; conversely, a channel is forced to remain when every solution excluding it is provably worse.
Useful6/10
Difficulty5/10
Novelty6/10
Unverified
2026
Build a neural feature-mixing block from a sparse exchange matrix and a fixed sequence of cluster-style mutations. Transpose-related channels share parameters through orbit tying, while positive multiplicative-additive updates are evaluated with log-sum-exp. The block is a concrete alternative to a dense MLP projection and can be tested at equal parameter count and compute.
Useful6/10
Difficulty5/10
Novelty8/10