Every idea extracted from recent arXiv mathematics papers — verified and unverified. Click an idea to open its full card; badges show the empirical verdict.
Replace a dense token-mixing, MLP, or low-rank-adapter projection with a Bernoulli-signed sparse matrix normalized by the paper's predicted operator scale. Estimate the active representation dimension and use the effective scale to detect regimes in which extreme sparsity is likely to cause unstable amplification or dead rows.
Replace a dense directional feature map with a finite stable-zonotope layer: a small set of large random vectors creates a hyperplane arrangement over the input sphere, while a cheaper small-jump residual gives continuous features inside each cell. The layer exposes an explicit compute-versus-resolution knob through the jump cutoff and can be used either as an embedding before an MLP or as a lightweight router for mixture-of-experts blocks.
Replace a dense unconstrained channel-mixing matrix with a differentiable product of exponentials of a few skew-symmetric generators and their iterated commutators. The resulting layer is exactly orthogonal, preserves feature norms, and can express rotations in directions not explicitly stored as independent parameters. This is especially suitable for residual MLP blocks, recurrent state transitions, and networks processing rotation- or pose-valued features.
Construct a sparse fixed orthogonal mixer by repeatedly applying pi/4 rotations to randomly matched pairs of feature coordinates. Place this mixer before top-k feature pruning, sparse projection, or activation quantization so that information is spread across coordinates without using a dense random matrix.
Build each nonlinear correction in an inverse neural operator from explicit bilinear products of learned operator features, following the inverse Born expansion instead of using an unconstrained pointwise MLP. Use a square activation to implement multiplication exactly, and truncate the interaction order so the model has a controllable polynomial structure.
Represent hierarchical or tree-structured hidden states on words over d symbols and replace a dense mixing layer by a noncommutative Toeplitz operator composed of shared word shifts. Coefficients are reused at every tree location, so the parameter count depends on maximum interaction depth rather than the number of nodes; an optional spectral penalty controls the amplification profile of finite-depth truncations.
Replace a dense multiresolution grid encoding for a coordinate MLP with a hierarchical sparse tensor-product encoding whose active cells are selected by local hierarchical surpluses. Combine anisotropic component grids with alternating binomial weights, then refine only regions whose encoded or prediction residual is large. This should preserve fine detail around localized structures while avoiding the exponential parameter count of a full grid.
Represent a 3D neural field using high-order polynomial coefficients attached to an adaptively refined tetrahedral mesh, with a small MLP predicting residual corrections from local coordinates. Refine only tetrahedra whose prediction, rendering, or PDE residual is large, and use globally ordered vertices so neighboring tetrahedra share identical face and edge coefficients without hanging-node constraint solves.
Separate low-support monomials, which involve only a few distinct input coordinates, from high-support monomials in a high-degree symmetric interaction layer. Compute the low-support orbit features exactly and prune, sample, or factorize the high-support tail, using the paper's cutoff scale as the initial sparsity rule.
Add a neural feature, adapter, or expert block only through the component of its outputs that is orthogonal to the span of all previously installed blocks. Quotient coefficient directions that produce nearly identical outputs with an SVD or pseudoinverse, so the new block contributes intrinsic representational dimensions instead of duplicating old features. The expected benefit is a smaller effective architecture and better-conditioned block expansion.
Replace isotropic variance control in a bottleneck or router with a spectral polarization penalty that drives each latent direction toward either variance 0 or variance 1. The intended result is an automatically selected active subspace: inactive coordinates can be pruned or quantized aggressively, while active coordinates retain information instead of being uniformly attenuated.
Replace a learned dense bilinear map with a structured family of signed orthogonal matrices. Given feature vectors y,z in R^n, produce r interaction features h_a = y^T H_a z / sqrt(n), where the H_a form a Hadamard/Clifford-like family; the resulting bilinear map has operator norm at most one when r is within the Hurwitz–Radon limit. Learn only channel projections and optional scalar gates around this fixed mixer.
Replace globally backpropagated hidden-layer losses with independent recurrent layers that receive bottom-up, top-down, and lateral inputs. Train each layer to assign low activity to correctly paired input-label examples and high activity to mismatched examples, then classify by selecting the label with the lowest accumulated surprise.
Use enumerated weighing matrices as sparse orthogonal channel-mixing operators inside MLPs or residual blocks. Their ternary entries reduce multiplication to signed additions, while exact orthogonality prevents amplification or attenuation of feature norms; a trainable fixed-support version can recover expressivity without giving up computational sparsity.
Represent a neural-network weight tensor by rank-one terms whose mode factors are selected from shared orthonormal bases, and impose the same basis alignment across tensor flattenings. During or after training, retain the largest coefficients to obtain a structured truncation analogous to truncated SVD. This should produce better-conditioned tensorized layers than unconstrained CP factors while preserving a directly controllable accuracy/compute tradeoff.
Replace heuristic magnitude pruning in a layerwise convex reconstruction problem with safe screening based on a perspective relaxation of the cardinality constraint. A channel is removed only when a lower bound for every solution containing that channel exceeds the loss of a feasible incumbent; conversely, a channel is forced to remain when every solution excluding it is provably worse.
Build a neural feature-mixing block from a sparse exchange matrix and a fixed sequence of cluster-style mutations. Transpose-related channels share parameters through orbit tying, while positive multiplicative-additive updates are evaluated with log-sum-exp. The block is a concrete alternative to a dense MLP projection and can be tested at equal parameter count and compute.
Replace a dense Gaussian or learned projection from dimension N to m with a normalized partial circulant projection generated by a single Gaussian vector. For K-sparse hidden states, the restricted-isometry guarantee predicts approximate norm preservation while reducing stored projection parameters from O(mN) to O(N). The projection can be evaluated with an FFT and should be combined with explicit top-k gating so that the sparse-input assumption is enforced.
Parameterize a large linear layer as a sum of binary tensor products, W = Σ_l A_l ⊗ B_l, and regularize a factor-level upper bound on its top-k singular-value sums. The bound controls all Ky Fan norms of W while requiring SVDs only of the small factors, making it suitable for tensorized MLP or attention projections.
Replace a dense order-p tensorized neural layer with a sparse random hypergraph tensor whose support probability is alpha = c/N^{p-1}. Assign independent zero-mean marks to active hyperedges, so every tensor moment scales as N^{-(p-1)} while the total variance entering each output remains controlled. The resulting locally tree-like computation graph should reduce parameters and multiply-adds without immediately destroying signal propagation.
Use the paper's correspondence between KAN splines and finite-element or isogeometric shape functions to build coordinate-separable tensor-product trial layers. Replace additive coordinate aggregation with a multiplicative contraction of one-dimensional spline expansions, yielding an explicit tensor-product basis without storing a dense multidimensional grid.
Replace a dense tensor-product positional encoding or first MLP layer with a hierarchical sparse-grid B-spline feature map. The network evaluates only localized basis functions indexed by multi-levels with bounded total level, reducing feature count while retaining high-order approximation for functions with mixed derivative regularity. The basis can initially be fixed and later fine-tuned jointly with the downstream network.
Compress a dense neural-network weight tensor with mode-wise selected fibers and a small sampled core, then convert the result into a Tucker layer for efficient execution. Select indices greedily using reconstruction residuals rather than random sampling, and fine-tune the compressed model to recover task accuracy.
Replace a dense block of a large neural-network weight matrix with an adaptively constructed cross approximation, but prevent premature termination using residual checks on O(m+n) additional diagonal-like entries. Accept a rank only after the probe residual has remained below tolerance for several consecutive iterations; otherwise continue adding pivots.