Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.
Replace Cox-de Boor evaluation of each cubic B-spline edge activation with its fixed truncated-power expansion. Normalize each scalar edge input to a bounded knot coordinate, evaluate the five shifted cubic positive-part terms in parallel, and contract them with the learned spline coefficients inside one fused kernel.
Replace ordinary topology-sensitive message passing with scalar-gated aggregation followed by an explicit correction that aligns local node states with a graph-wide consensus component. The correction should make node embeddings less sensitive to line or edge removals while preserving local information needed for prediction. This is suitable for graph neural networks and graph-based world models exposed to changing graph sizes or sparsity patterns.
Replace vector-valued Hopfield neurons by SU(d)-valued latent states and construct Hebbian couplings from matrix memories. Recall is performed by iterating toward the dominant eigenmode of the induced lifted coupling operator, with each iterate projected back onto SU(d); the larger matrix representation should reduce random crosstalk and increase critical memory capacity.
Construct a spatial recurrent network whose local vector hidden state has two stable attractors and whose neighbor coupling is diffusive. Train or constrain the network so that the desired attractor invades the undesired one with a controlled positive front velocity, rather than relying on a scalar class-frequency variable that can erase depletion and interface structure.
Replace direct learning of a highly cancelling signed observable with a quotient-space model over symmetry orbits of inputs. Predict a physically constrained baseline for each family and use an LSTM or set/graph encoder only for the residual many-body correlation, then aggregate family predictions with known signed weights instead of forming a noisy sample-level ratio.
Give a single spiking layer a persistent vector-valued apical compartment that stores the current online linear predictor for the task. On each labeled context pair, its subthreshold state performs a leaky LMS update; on the query, the state is read without updating, allowing in-context adaptation without attention or inference-time synaptic plasticity.
Replace a fragile learned similarity score for continuous object coordinates with a projection residual against a learned signal subspace. Candidate coordinates are represented by normalized Fourier or positional feature vectors, and the score is the fraction of feature energy outside the estimated subspace. The score remains useful even when the estimated subspace is adversarially rotated, because the perturbation is controlled directly by a sine-theta distance rather than by assumptions about…
Use a spherical-design codebook and the paper's polar slack factorization to create a nonnegative geometric interaction bias for attention or expert routing. The resulting kernel is generated by a rank-one term and a rank-at-most-d term, and entries close to zero can define a structured sparse mask instead of relying only on learned top-k selection.
Precondition activation or cache blocks with a fixed product U = A Sigma B of orthogonal transforms and a random signed permutation before quantization or coordinate sampling. The random permutation makes the product incoherent, so energy is less concentrated in a few coordinates and lossy compression should introduce less worst-case distortion.
Replace fixed random Fourier features in a coordinate network or PINN with a small complex Fourier dictionary whose propagation directions are learned from a weighted residual. Given directions, solve the linear feature coefficients exactly or by ridge regression, and optimize only the directions in the outer loop. This should represent low-directional-complexity fields with fewer features and avoid wasting gradient updates on coefficients that can be fitted analytically.
Make a neural network predict a vector potential rather than a magnetic or velocity field, then obtain the physical vector field with a fixed differentiable discrete curl. The reconstructed field satisfies the discrete divergence-free constraint exactly, eliminating divergence-penalty tuning and preventing constraint drift during long rollouts.
Make a neural network predict a positive Gaussian-mixture representation of the distribution function rather than independent values on a momentum grid. Use the mixture parameters inside a differentiable Boltzmann collision operator, so training directly enforces the interaction mechanism and exposes the relaxation spectrum responsible for ballistic-to-hydrodynamic crossover.
Evaluate weak residuals against a bank of periodic trigonometric test functions using FFT projections instead of repeated pointwise quadrature or output automatic differentiation. Frequency truncation and mode weighting provide a direct way to control the spatial scales enforced during neural PDE training.
Construct a recurrent or state-space layer as a skew product: an expanding bounded feature coordinate drives a linearly contracting hidden state. Constrain the hidden transition matrix A to have spectral radius below one, and monitor the predicted transition ell times the absolute determinant of A equals one: below it, hidden trajectories should occupy a thin or fractal set, while above it they should have substantially higher-dimensional state coverage without losing local contraction.
Run the same neural decoder over several algebraically equivalent augmented graphs and aggregate their variable-level predictions. Each graph exposes different cycle structure and message routes, providing structured architectural diversity rather than ordinary random-seed ensembling.
Add auxiliary constraint nodes generated from linear combinations of existing constraints, creating a new message-passing graph while preserving the original feasible error set. Use a neural BP layer on the augmented graph so auxiliary nodes provide alternate paths around harmful cycles without changing the target constraints.
Initialize a recurrent or state-space transition matrix with weak Wigner noise plus a shared cumulative-sum correlation structure. Increasing the correlation strength produces recurrent eigenmodes one at a time at analytically predicted BBP thresholds, yielding a controllable hierarchy of short- and long-memory modes. The matrix should then be globally rescaled or constrained so that all active modes remain inside the desired stability radius.
Replace independent pairwise feature matching across augmented or multimodal views with jointly estimated soft permutation matrices constrained to agree through cycles. The paper's multi-view result suggests that independent copies can cross a correspondence-recovery threshold even when every individual pairwise matching is statistically non-informative. In a neural network, this can provide cleaner token, patch, object, or cell alignment targets and can be used either as a differentiable…
Replace the unconstrained transition of a recurrent or state-space neural network with a DMDc-initialized linear latent transition plus a learned nonlinear residual. Estimate the transition from a short warm-up dataset using Hankel delay coordinates, retain eigenmodes with decay rates near the unit circle for long-term memory, and let the neural residual model dynamics not explained by the linear backbone. This should make long-horizon prediction and slowly varying signals easier to learn while…
Replace several fixed message-passing layers with an implicit graph layer whose state is the solution of a nonlinear flow equilibrium. Learn monotone edge laws from endpoint features, solve for node potentials with damped chord-Newton steps, and use the resulting edge flows or potentials as the layer output. Monotonicity and the Laplacian Jacobian provide a principled stability mechanism while retaining sparse graph computation.
Replace the dense decoder of an overcomplete activation autoencoder with a fixed left-d-regular expander mask and learn only the nonzero decoder values. Tie the encoder to the transpose of this sparse decoder, preserving the activation dimension m, latent width n, and TopK sparsity k while reducing learned dictionary parameters from mn to dn.
Replace magnitude-based channel or expert pruning with a subset-selection objective that maximizes the weakest direction in the candidates' activation span. Relax the binary mask to continuous gates, optimize an entropic soft minimum eigenvalue, and round the gates to retain a fixed number of channels or experts. This should preserve diverse representations and reduce redundant feature directions.
Construct a sparse radius graph over input samples or learned node embeddings, compute its cycle space, and remove the subspace generated by sufficiently short cycles. Feed the remaining quotient-cycle coordinates or Betti-rank estimate to a graph neural network as a global topological feature, or use them to guide pooling so that local redundant loops are collapsed while global loops are retained. The paper predicts that the threshold L approximately equal to |log r| graph hops is the critical…
Train a network to predict the context-dependent observation matrix rather than the latent inverse parameters themselves, then compute the latent parameters with a differentiable ridge-regression solve. This gives one model that can assimilate arbitrary observation vectors, exposes the conditioning of the inverse problem, and avoids forcing an MLP to learn the entire map from observations to parameters.