Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.
Replace a single smooth neural vector field with a finite collection of smooth subnetworks selected by learned affine hyperplanes. The architecture exposes switching geometry directly, allowing it to represent friction-like or threshold dynamics without approximating discontinuities using excessively steep activations.
Replace a conventional linear decoder in an autoencoder or latent state-space model with an explicit quadratic manifold decoder, allowing a small latent vector to represent curved and transport-like state trajectories. Add a dynamics-aware invariance loss that penalizes the discrepancy between the time derivative of the quadratic manifold and the neural dynamics evaluated on that manifold.
Attach an SU(2) transport matrix to every directed edge of a graph neural network and penalize nontrivial plaquette holonomies instead of penalizing individual edge transformations. The regularizer is invariant to arbitrary local changes of latent representation frame, encouraging path-consistent relational features without requiring all edges to share one global coordinate system.
Replace a dense Haar or Gaussian random projection with a streamed product of random two-coordinate rotations followed by coordinate subsampling. The transform is exactly orthogonal before subsampling, requires only a list of rotation triples, and the paper's pseudo-mixing result predicts that degree-two statistics relevant to norm preservation and Johnson–Lindenstrauss embeddings become Haar-like after only O(n polylog(n)) rotations.
Replace a conventional whitening transform with a constrained whitening layer that minimizes cross-channel covariance while requiring every output channel to remain correlated with its designated input channel by at least a threshold \(\rho_{\min}\). The layer exploits the orthogonal freedom in whitening to find a rotation that preserves channel identity instead of arbitrarily mixing features. It can be inserted before an MLP, convolution, or attention projection and compared directly against…
Insert a linear Johnson–Lindenstrauss bottleneck around a set of jointly processed representations, choosing its width from the sharp finite-set dimension bound rather than from the model's nominal hidden size. The projection should preserve pairwise distances between tokens, patches, or retrieved items, allowing a downstream attention or MLP block to operate at lower width while retaining the geometry relevant to similarity computations.
Replace a wide collection of interchangeable near-zero branches with a module whose output is explicitly a quadratic form in the branch-weight Gram matrix. The module preserves the paper's leading-order behavior while making the relevant collective variable explicit and allowing low-rank parameterizations.
Parameterize an entropic OT cost only in directions that can change the transport plan, removing row-plus-column potential directions that are invisible because of OT gauge invariance. Whiten the remaining feature coordinates using their empirical covariance, producing an OT layer whose identifiable parameters have substantially more uniform sensitivity.
Use trajectory sensitivities to remove neural units or parameter groups whose effects are redundant over the available data support. A parameter group is pruned when its Fisher contribution is small or its sensitivity is nearly collinear with other groups, producing a compact neural ODE without relying only on parameter magnitude.
Replace deterministic binary-tree pooling or hierarchical feature aggregation by a stochastic merge that chooses either elementwise addition or elementwise minimum. The mixing probability p controls whether zero or sparse states proliferate or disappear, with a predicted absorbing-state transition at p = 1/2.
Replace blind width or rank expansion with a residualized candidate-selection procedure for low-rank adapters or expert banks. Each candidate block is measured after removing the component already represented by the incumbent network, filtered for numerical conditioning, and accepted only when its projected response produces sufficient loss-relevant gain. This should produce smaller adaptive networks than adding candidates by fixed order or magnitude heuristics.
Add an online receptive-field expansion monitor to a graph neural network and use it to gate message-passing depth or invoke graph pooling. For a sampled node set F and propagation neighborhood K, continue fine-scale propagation only while the growth ratio |KF|/|F| is close to one; when it is persistently expansive, replace further propagation with pooling, local attention, or long-range skip messages. This transfers the paper's Følner-versus-paradoxical mechanism into an architecture-level…
Convert each persistence diagram produced from an input, intermediate feature map, or graph filtration into a discretized persistence landscape and feed it to an MLP or concatenate it with ordinary neural features. Unlike a variable-size list of birth-death pairs, the landscape has a fixed tensor shape and is provably nonexpansive with respect to the diagram Wasserstein distance.
Add a conditional-law head that maps a compact representation of an initial distribution and a shared-noise trajectory to a Gaussian mixture, then computes downstream predictions as analytic expectations under that mixture. This can replace expensive particle rollouts or particle pooling in stochastic world models and conditional diffusion systems while retaining multimodality.
Use predicted covariance reduction as a differentiable gate for selecting tokens, views, sensors, or retrieved demonstrations. The gate favors inputs with high expected information gain while accounting for acquisition cost, turning attention and data collection into active observability optimization.
Replace the single global MoE capacity factor with expert-specific capacity reservations chosen from a small reliability menu. Experts with highly variable or operationally important token loads receive larger robust buffers, while predictable experts run closer to their mean load. This should reduce token dropping and padding waste simultaneously, especially under distribution shift or bursty routing.
Replace part of a CNN or continuous-depth feature block with two coupled feature fields. One field is transported up gradients of the other through a conservative cross-gradient flux, creating adaptive spatial organization that ordinary diffusion or symmetric convolution cannot produce. The coupling strength and dominant wavelength are controlled by a directly testable linear-instability boundary.
Construct the sparse transition matrix and input projection of a recurrent or state-space layer so that every hidden-state row is covered by a matching in the controllability core. This prevents hidden directions from becoming unreachable from the input sequence, especially in multi-input systems and across a distribution of transition matrices or task conditions.
Represent a hybrid trajectory with one neural module per known dynamical phase rather than a single network spanning all phases. Feed the predicted terminal state of phase r directly as the initial state of phase r+1, so continuity is satisfied by construction instead of by a soft interface penalty. This should improve learning near abrupt changes and remove an otherwise poorly conditioned loss-weight tradeoff.
Replace an unconstrained image denoiser or refinement block by a gradient step on an input-convex neural potential. The resulting map has a verifiable nonexpansiveness guarantee when the potential is convex and its gradient is sufficiently smooth, reducing error amplification across repeated applications and making the module safer under distribution shift.
Replace a Markovian recurrent update with an MPS-valued temporal influence state that couples adjacent pairs of memory sites, mimicking the paper's CDU3 two-column construction. The hidden state retains structured correlations across multiple past time steps while computation remains linear in sequence length and polynomial in the bond dimension, rather than exponential in the memory horizon.
Train a cheap shared multi-task probe briefly, extract one semantic embedding per task, and use density-based clustering to determine which tasks should share a neural trunk. After clustering, replace the globally shared trunk by one trunk per discovered cluster, with task heads remaining separate; this preserves cooperation among related tasks while isolating destructive task interactions.
Replace deterministic LoRA importance scores with one-sided tests of whether each rank-one update has population contribution at least a user-selected threshold. Maintain empirical contribution samples during fine-tuning, estimate their uncertainty, and prune the components with the weakest statistical evidence while respecting the target rank budget. The method should avoid deleting components merely because their latest minibatch gradient was small.
Add a differentiable orientation-pooling layer after steerable filters or an orientation-bin expansion. It aggregates unoriented line evidence using doubled-angle vectors, so a feature at angle θ is identical to one at θ+π, while symmetric orientations cancel naturally instead of producing an arbitrary mean angle. Feed the network both the Cartesian order parameter and its magnitude-based confidence.