Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.
Replace Cox-de Boor evaluation of each cubic B-spline edge activation with its fixed truncated-power expansion. Normalize each scalar edge input to a bounded knot coordinate, evaluate the five shifted cubic positive-part terms in parallel, and contract them with the learned spline coefficients inside one fused kernel.
Construct a residual network from independently attachable modules, but permit only a number of modules whose aggregate feedback gain lies inside a delay-dependent admissible interval. Estimate deployed end-to-end latency and each module's local Jacobian gain, then reject or bypass additional modules when the predicted delayed-loop stability boundary is crossed. This turns variable-width or depth scaling into a falsifiable control problem rather than an empirical choice.
Replace unrolled autodiff through an ordered block-implicit neural layer with a custom reverse sweep that solves one small transposed local system per forward block update. The backward computes the exact gradient of the executed finite-depth solver while avoiding a global Jacobian and retaining only compact block information.
Replace the uniform or power-law convolution in a recurrent or state-space layer by a Gaussian q-binomial fractional kernel with learnable order alpha and deformation q. The parameter q controls a concrete memory-localization transition: q close to 1 gives classical fractional power-law memory, whereas q<1 produces exponentially localized memory and should reduce long-horizon gradient interference and truncation cost.
Represent a large linear map acting on a Cartesian 3D grid and multiple physical channels as a TT-matrix, while retaining separate TT blocks for channel couplings that have different semantics. Apply the layer by sequential contractions with TT cores rather than materializing a dense matrix or a full 3D convolution kernel. Rank truncation provides an explicit accuracy-versus-memory knob and can be applied after optimizer updates.
Replace full PSD self-attention with a pivoted Cholesky/Nyström approximation whose landmarks are sampled from the unexplained diagonal mass. Tokens with large residual self-similarity are more likely to become landmarks, so the rank budget is spent on difficult regions rather than uniformly selected tokens.
Build a label-free router for a finite library of neural operators by estimating one shared physical target response from an anchor prediction and using it to rank every candidate through inner products with candidate differences. The method avoids running a full residual-based diagnostic independently for every expert and can be used either to select the best expert or to form a corrected weighted combination.
Replace a trainable shallow MLP hidden layer by a frozen bank of smooth sigmoid ridge functions and train only a linear output head. Choose the feature count and parameter sampling regime using the theorem's explicit dependence on input dimension d, target regularity k, evaluation norm m, and confidence delta. The construction is especially appropriate for smooth regression, scientific surrogate models, and PINNs, where derivatives of the network output are part of the loss.
Replace per-particle message evaluation in a point-cloud or particle-based neural layer with exact box moments. Particles inside a box are compressed into a fixed tensor of monomial sums, and every query in that box evaluates the same piecewise-polynomial interaction from those moments, reducing work from particle-query pairs to particles plus occupied boxes.
Attach a polyharmonic spline decoder to a coordinate MLP or use it as a standalone neural-field output head over a large set of spatial anchors. The decoder represents the output as a low-degree polynomial trend plus a PHS kernel expansion, while FMM evaluates all anchor-to-query interactions in approximately linear or near-linear cost. When coefficients must be fitted or periodically recalibrated, solve the constrained interpolation system with projected conjugate gradients and a sparse…
Evaluate weak residuals against a bank of periodic trigonometric test functions using FFT projections instead of repeated pointwise quadrature or output automatic differentiation. Frequency truncation and mode weighting provide a direct way to control the spatial scales enforced during neural PDE training.
Replace dense continuous action search during neural-controller verification with a finite set of representative inputs induced by affine pieces of the interval neural dynamics. This makes safety checking parallel over state cells and candidate actions, enabling much cheaper certification or repeated safe-set updates.
Replace a stack of local message-passing layers by a fractional spectral graph filter implemented through a small bank of sparse shifted Laplacian solves. The fractional exponent controls how strongly the layer mixes information across graph distances, while rational approximation avoids dense eigendecomposition and supports efficient differentiation through iterative linear solvers.
Replace sequential upward message passing in a tree-structured neural module with rake–compress contraction of quadratic latent-state messages. Each node stores a quadratic value function and each edge stores a linear transition or coupling triple; leaf elimination and unary-node compression are implemented as batched Schur complements, followed by a reverse pass that reconstructs node latents and edge outputs. The layer is exact for Gaussian or quadratic latent models and remains…
Replace dense attention between tokens on opposite sides of a one-dimensional boundary or segment split with a dyadic low-rank approximation of a Cauchy/Hankel distance kernel. Each distance-scale block uses O(log(1/\varepsilon)) features, and the number of active scales grows only logarithmically with context length after discarding a narrow boundary layer. This is especially suitable for a relative-position attention branch or state-space-like long-range branch, rather than arbitrary…
Build an SPD classifier and residual head directly from Cholesky factors, using lower-triangular differences and matrix-power terms instead of generic eigendecomposition-based logarithm operators. This retains covariance geometry while making positive-definiteness automatic and backpropagation more numerically stable for minibatch training.
Replace Lorentz-hyperboloid tensors with proper-velocity tensors whose spatial coordinates can be transformed by standard Euclidean affine layers and activations. Reconstruct the Lorentz time coordinate only at manifold boundaries, preserving the hyperbolic representation while avoiding repeated projection, normalization, or fragile exponential-map calculations.
Split hidden dynamics into relaxation bands when the Jacobian spectrum has a gap, evolve each band with its own timescale, and retain an explicit cross-band exchange term. This yields a principled dual-timescale RNN or SSM rather than choosing fast and slow branches heuristically.
Replace dense query-key attention with an adaptive cross approximation constructed from selected query and key pivot tokens. At each rank, choose the pivot pair that removes large estimated residual energy, update the residual by a rank-1 cross correction, and stop when the residual estimate reaches a target tolerance. The resulting factorization computes approximate attention using a small number of landmark interactions while adapting to the actual token distribution.
Train a residual network on a coarse depth mesh, estimate a dual-weighted residual for every layer interval, and insert new layers at intervals with the largest estimated contribution to objective error. This replaces uniform depth expansion or expensive neural architecture search with targeted refinement driven by both forward-dynamics error and downstream loss sensitivity.
Replace a weight-tied residual or neural-ODE stepper with an explicit Runge–Kutta method satisfying the reused-last-stage conditions. The final derivative is evaluated at the exact endpoint and becomes the first derivative of the next step, saving one expensive neural-vector-field call per step while preserving the designed integration order.
Represent a high-order feature tensor as a tensor train and replace a dense global feature transform by a truncated polynomial in a learned nearest-neighbor operator. The block computes a short Krylov expansion, p_m(A)x = sum from k=0 to m of c_k A^k x, compressing back to a fixed TT rank after each operator application; locality is intended to prevent rank growth from scaling with the total number of tensor sites.
Replace the usual gradient-descent or conjugate-gradient data-fidelity step in an unrolled reconstruction network with an exact Woodbury proximal layer for grouped multiplexed measurements. The layer can be inserted between learned denoising blocks and should provide stronger measurement consistency at a fixed number of unrolled stages, while avoiding inner iterative linear solves.
Replace a dense graph-attention or token-mixing matrix by a resolvent-like interaction operator and truncate it to graph neighborhoods whose radius is selected from an estimated spectral gap. Unlike fixed-window sparse attention, the sparsity level is tied to a measurable stability parameter and has an explicit exponential tail criterion.