Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.
Replace Cox-de Boor evaluation of each cubic B-spline edge activation with its fixed truncated-power expansion. Normalize each scalar edge input to a bounded knot coordinate, evaluate the five shifted cubic positive-part terms in parallel, and contract them with the learned spline coefficients inside one fused kernel.
Construct a residual network from independently attachable modules, but permit only a number of modules whose aggregate feedback gain lies inside a delay-dependent admissible interval. Estimate deployed end-to-end latency and each module's local Jacobian gain, then reject or bypass additional modules when the predicted delayed-loop stability boundary is crossed. This turns variable-width or depth scaling into a falsifiable control problem rather than an empirical choice.
Replace pointwise prediction of the next field with prediction of a learned flux followed by a discrete divergence. Combine Fourier spatial mixing with a causal temporal kernel over the recent resolved-history slab, so the model learns finite-memory closure effects while preserving local conservation exactly under periodic or compatible boundary conditions. The architecture should reduce spurious mass drift and improve autoregressive rollout stability on coarse-grained PDE data.
Replace hard clipping or post-hoc asymmetric saturation with a dynamic output state that remains inside a prescribed asymmetric interval. A neural network emits a command uc, while the realized output u evolves through the APIR vector field, producing bounded actions, temporal smoothing, and gradients that remain available in the interior.
Build a latent continuous-time neural model with dynamics \(\dot{z}=Az+f_\phi(z)\), where \(f_\phi\) is known, separately computed, or frozen, and \(A\) is learned exclusively from the derivative residual after subtracting \(f_\phi(z)\). Parameterize \(A\) with a truncated SVD or low-rank factorization so its eigenvalues directly predict local stability and long-horizon growth.
Replace unrolled autodiff through an ordered block-implicit neural layer with a custom reverse sweep that solves one small transposed local system per forward block update. The backward computes the exact gradient of the executed finite-depth solver while avoiding a global Jacobian and retaining only compact block information.
Compute an inner approximation of the states from which a neural controller can keep the plant inside a prescribed safe domain indefinitely, then use the resulting regulation map as a safety shield around the network. At each state, the network proposes an action, but the shield projects or replaces it with an action certified to remain in the invariant set.
Use discovered infinitesimal generators to create small continuous transformations of hidden states and force a neural predictor to commute with those transformations. This converts symmetry discovery into self-supervised augmentation without prespecifying a group, canonical coordinates, or hand-designed equivariant layers.
Replace full-state quantized write-back in a deep low-bit residual stack with quantized increment error feedback. The residual branch quantizes the proposed increment after adding the previous carry, while the carry stores the exact discrepancy; this makes the total error telescope instead of accumulating approximately once per layer.
Attach a hard control-barrier-function quadratic-program safety filter to a neural policy, but solve the filter with operator splitting and differentiate through its fixed-point map using projection Jacobian-vector products. The network learns the nominal action and task objective end to end, while the deployed action remains the feasible filtered action rather than an unconstrained penalty-based approximation.
Replace the uniform or power-law convolution in a recurrent or state-space layer by a Gaussian q-binomial fractional kernel with learnable order alpha and deformation q. The parameter q controls a concrete memory-localization transition: q close to 1 gives classical fractional power-law memory, whereas q<1 produces exponentially localized memory and should reduce long-horizon gradient interference and truncation cost.
Replace a single neural representation of a rapidly oscillatory PDE solution by a macroscopic network plus an explicitly oscillatory corrector network. Feed the network both the slow coordinate $x$ and fast coordinate $y=x/\varepsilon$, and train the resulting composite field in a variational energy objective. This targets the paper's scale-robust approximation bound rather than forcing the optimizer and finite sample set to discover oscillations of wavelength $\varepsilon$.
Modify a point-cloud message-passing or neural-operator layer so that scalar gradients, vector features, and vector-to-vector interactions are computed only in the estimated tangent plane of the surface. Projecting both feature values and derivative directions prevents the network from using arbitrary ambient-space normal directions and should improve transfer across differently embedded but intrinsically similar surfaces.
Augment a 1D neural operator or transformer with explicit tokens for detected discontinuities. Advance each front analytically using the local Rankine–Hugoniot speed and train the network only to reconstruct smooth regions and the residual caused by source terms and grid resolution.
Build a neural operator from frozen ambient mechanism blocks and a geometry-specific algebraic constraint adapter. The adapter parameterizes all outputs in the affine set satisfying sampled linear constraints exactly, so the network never produces boundary-violating states and does not require a penalty coefficient or post-step projection.
Replace a standard recurrent state update or finite-order SSM filter with a causal relative-history operator using a weakly singular kernel k(s)=s^{p-1}m(s), where 0<p<1. The resulting layer retains information over a power-law range of timescales and introduces tunable frequency-dependent phase and attenuation, while remaining implementable through a small bank of exponentially decaying states.
Represent a large linear map acting on a Cartesian 3D grid and multiple physical channels as a TT-matrix, while retaining separate TT blocks for channel couplings that have different semantics. Apply the layer by sequential contractions with TT cores rather than materializing a dense matrix or a full 3D convolution kernel. Rank truncation provides an explicit accuracy-versus-memory knob and can be applied after optimizer updates.
Construct an efficient recurrent or state-space layer whose impulse response follows Mittag-Leffler relaxation instead of a single exponential. A bank of stable diagonal state channels approximates the long power-law tail, allowing the layer to retain information over widely separated timescales with only \(K\) states per feature.
Replace full PSD self-attention with a pivoted Cholesky/Nyström approximation whose landmarks are sampled from the unexplained diagonal mass. Tokens with large residual self-similarity are more likely to become landmarks, so the rank budget is spent on difficult regions rather than uniformly selected tokens.
Build a label-free router for a finite library of neural operators by estimating one shared physical target response from an anchor prediction and using it to rank every candidate through inner products with candidate differences. The method avoids running a full residual-based diagnostic independently for every expert and can be used either to select the best expert or to form a corrected weighted combination.
Replace a trainable shallow MLP hidden layer by a frozen bank of smooth sigmoid ridge functions and train only a linear output head. Choose the feature count and parameter sampling regime using the theorem's explicit dependence on input dimension d, target regularity k, evaluation norm m, and confidence delta. The construction is especially appropriate for smooth regression, scientific surrogate models, and PINNs, where derivatives of the network output are part of the loss.
Replace per-particle message evaluation in a point-cloud or particle-based neural layer with exact box moments. Particles inside a box are compressed into a fixed tensor of monomial sums, and every query in that box evaluates the same piecewise-polynomial interaction from those moments, reducing work from particle-query pairs to particles plus occupied boxes.
Freeze a wide neural spatial dictionary, then compress and whiten it using the quadrature mass matrix before solving for output coefficients or latent PDE states. The retained basis removes feature directions that are numerically invisible or nearly dependent under the actual domain discretization, while preserving the represented function space up to the chosen SVD rank.
Make a neural network predict a vector potential rather than a magnetic or velocity field, then obtain the physical vector field with a fixed differentiable discrete curl. The reconstructed field satisfies the discrete divergence-free constraint exactly, eliminating divergence-penalty tuning and preventing constraint drift during long rollouts.