Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.
Replace dense attention over structured object tokens with attention over role-filler tensor-product representations. A learned query specifies both a role and a filler, retrieves objects matching that binding, extracts a target role, and rebinds the extracted filler into an output object.
Replace the usual sum of pairwise modality similarities with a higher-order score based on the coordinatewise Hadamard product of all normalized modality embeddings. For modalities indexed by i=1,...,m, score a tuple using s(x_1,...,x_m)=\omega^\top(\bar g_1(x_1)\odot\cdots\odot\bar g_m(x_m)), where \omega is learned and \odot is coordinatewise multiplication. This adds explicit m-way interactions without concatenating raw features or introducing a joint encoder.
Augment a 1D neural operator or transformer with explicit tokens for detected discontinuities. Advance each front analytically using the local Rankine–Hugoniot speed and train the network only to reconstruct smooth regions and the residual caused by source terms and grid resolution.
Replace an isotropic local mixing layer with a kinetic layer that smooths features in x and transports them in y along the characteristic direction x. The layer should be useful for phase-space data, learned simulators, and world models in which positions or transported quantities evolve through coupled drift and diffusion rather than independent Euclidean motion.
Use a learned scalar ordering function to turn a symmetric local Gaussian graph kernel into a directed, row-stochastic message-passing operator. The asymmetric tilt lets neighboring nodes communicate preferentially along an inferred dynamical direction, while the Gaussian factor retains locality and diffusion-like smoothing.
Parameterize an entropic OT cost only in directions that can change the transport plan, removing row-plus-column potential directions that are invisible because of OT gauge invariance. Whiten the remaining feature coordinates using their empirical covariance, producing an OT layer whose identifiable parameters have substantially more uniform sensitivity.
Freeze a randomly initialized single-layer transformer and use a constructed soft prompt to make its attention weights equal Gaussian-kernel weights over support examples. The resulting model performs Nadaraya-Watson regression in one forward pass, so task adaptation stores prompt tokens rather than modifying network weights. Prompt length becomes the number of kernel centers, while hidden dimension and prompt norm determine whether the required logits can be represented accurately.
Use predicted covariance reduction as a differentiable gate for selecting tokens, views, sensors, or retrieved demonstrations. The gate favors inputs with high expected information gain while accounting for acquisition cost, turning attention and data collection into active observability optimization.
Replace globally unique node IDs with a small discrete alphabet labeling whose ordered local neighborhood words uniquely identify nodes. Feed these separating words to a GNN or graph transformer as positional features, preserving node distinguishability with bounded local receptive fields.
Replace a portion of quadratic key-value attention or an external episodic table with a per-sample matrix fast memory updated by rank-one delta corrections. The memory directly learns a linear key-to-value map and can be carried across sequence segments, providing cheap online adaptation with constant state size per head.
Replace ordinary pairwise attention similarity by an affinity averaged over transformed keys or values. The resulting attention is invariant to the group action on either input and avoids requiring the network to learn identical attention patterns for every rotated or transformed copy.
Replace independent pairwise feature matching across augmented or multimodal views with jointly estimated soft permutation matrices constrained to agree through cycles. The paper's multi-view result suggests that independent copies can cross a correspondence-recovery threshold even when every individual pairwise matching is statistically non-informative. In a neural network, this can provide cleaner token, patch, object, or cell alignment targets and can be used either as a differentiable…
Augment every graph node with a vector of rooted walk and motif densities rather than relying only on degree or Laplacian positional encodings. This should distinguish nodes or communities with identical expected degree but different connectivity profiles, especially in equal-degree stochastic block models and graphs with locally heterogeneous structure.
Use separate measurement-domain and target-domain token sets so a network can infer a field on one spatial domain from sparse observations on another in one forward pass. The same decoder can answer arbitrary target query points, avoiding an optimization loop for each inverse instance.
Add a bank of temporal branches whose synapses apply heterogeneous delays before coincidence pooling. A branch selectively responds to an ordered pair or short sequence when the delay difference compensates the sequence interval, while low delay dispersion makes it primarily an order-invariant event detector. The branch threshold creates sparse, interpretable activations that can replace part of a recurrent or attention-based temporal module.
Insert a differentiable reaction-diffusion layer that converts dense token or pixel features into sparse, spatially coherent routing masks. Two competing orientations form complexes through conserved monomer reservoirs, so local assignments can cluster while opposite assignments mutually exclude one another instead of independently activating at the same location. The layer can be used as a soft-to-hard MoE router, attention-mask generator, or object-part grouping module.
Replace independently normalized attention or routing weights with an entropic doubly stochastic transport plan, while choosing its regularization ε using the paper's explicit statistical-stability bound. Increase ε when residual inversion or minibatch fluctuations are amplified, and decrease it only when the estimated bound permits sharper assignments.
Replace ordinary absolute positional embeddings with coordinates on a learned flat torus and use dual-lattice Fourier characters as positional features. Control the covariance of the coordinate fundamental domain so that the paper's inequality guarantees a lower bound on the smallest nonzero positional frequency, preventing the learned periodic coordinate system from developing arbitrarily weak or nearly constant modes.
Replace unconstrained attention score vectors by normalized SU(2) coherent-state responses of a positive operator on an (N+1)-dimensional spin space. Each query produces a smooth bounded response over a fixed spherical grid, while values are aggregated normally. The coherent-state kernel imposes geometric structure and exposes a controllable concentration parameter N.
Use a fixed spherical t-design as the direction codebook for a directional attention or feature-aggregation module instead of independently sampled random directions. Equal weights provide exact zero mean and isotropic second moments, while exactness for spherical polynomials up to degree t reduces directional aliasing and seed-dependent anisotropy.
Replace the usual top-eigenvector positional encoding in a graph neural network with a density-selected spectral subspace. The selector explicitly searches below the leading eigenvectors, where community information may survive after latent geometric modes have consumed the largest eigenvalues. The selected coordinates can be concatenated to node features or used as a bias in graph attention.
Replace independent Gaussian attention noise or unconstrained token routing with a directed-polymer path distribution over positions and layers. The router aggregates exponentially many monotone paths through temporally correlated random edge scores, producing heavy-tailed but spatially coherent routing and preventing attention from collapsing onto a single token. The paper's t^{2/3} wandering and t^{1/3} free-energy fluctuations become measurable diagnostics and tunable targets rather than…
Compute a compact multiscale interaction signature between colored point clouds and append it to a point-cloud or multimodal neural network as a learned interaction token. The signature captures separated, overlapping, and higher-order enclosing configurations while remaining invariant to rigid transformations.
Augment standard Transformer positional embeddings with coordinates generated by the paper's signed-base digit expansion. Previous binary digits determine the sign and scale of later contributions, while a two-state Markov chain controls correlations between digits. This supplies multiscale positional structure using a small number of transition and base parameters.