Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.
Project each FFN residual update onto the tangent space of the current token residual direction before adding it to the stream. This preserves the component that changes representation direction while suppressing norm-only motion, which may reduce residual-norm drift and aggregation-induced representation collapse.
Replace a conventional signed-graph message-passing layer with two coupled feature channels: a fixed channel invariant under switching and an anti-fixed channel that changes sign under switching. Unsigned aggregation updates invariant features, while signed aggregation updates anti-invariant features, implementing the paper's sphere-plus-involution representation at the hidden-state level.
Replace an unconstrained recurrent or residual transition with a block-structured contractive realization whose input-dependent multiplier is a direct sum of bounded branches. The resulting layer has a built-in non-expansive energy bound on the chosen normalized input domain, while still allowing different branches to respond to different coordinates or feature groups.
Store a finite library of successful robot configurations or action-conditioned waypoints and construct a smooth soft minimum of their distances. Use the negative distance gradient as a structured action prior, add a learned residual policy, and pass the combined action through a quadratic-program safety layer. This gives a neural controller an explicit attraction basin toward demonstrated solutions while preventing violations of known state constraints.
Construct a routed neural state update from a collection of branch maps whose inverse-style refinement operators are uniformly contractive. The discrete routing sequence acts as an expandable symbolic code, while contraction makes the continuous state associated with a long routing history insensitive to initialization and earlier perturbations. Use a finite active top-\(K\) set during training, but retain an expandable branch table so the model can represent increasingly complex or rare modes.
Replace dense spatial attention on a 2D feature grid by attention from every non-landmark token to a small periodic set of landmark tokens. Choose landmarks using the paper's residue construction so that every non-landmark token has at least one landmark exactly k Manhattan steps away. Use this guaranteed edge as a sparse global-information route, combined with a cheap local convolution or residual path for accuracy.
Replace an unconstrained quadratic interaction between channel derivatives with a learnable combination of Lorentzian and antisymmetric null forms. For wave-equation surrogates, this enforces exact cancellation when two interacting features have parallel null directions, suppressing resonant derivative products that otherwise cause unstable long-horizon rollouts.
Represent the predicted solution as $u_{\theta}(x)={\rm d}_{\Omega}(x)^s v_{\theta}(x)$, where $v_{\theta}$ is an unconstrained neural network and ${\rm d}_{\Omega}$ is the distance to the boundary. This builds the fractional Dirichlet boundary layer into the architecture and leaves the network to learn the smoother quotient $u/{\rm d}_{\Omega}^{s}$, which the paper proves extends Hölder-continuously to the boundary when the reaction has sufficient integrability.
Build a sparse expert cascade whose router uses empirical progress, uncertainty, and compute cost to construct an efficient frontier of experts. Instead of always choosing the expert with the largest immediate gain per FLOP, route different confidence states through an ordered sequence of frontier experts, allowing cheap high-variance experts early and safer or more cost-effective experts near the final decision.
Apply the paper's mechanism-contrast idea to ReLU decoders by requiring each piecewise-affine branch to produce a detectable and distinctive change across at least one activation boundary. Penalize branches with vanishing Jacobian jumps or nearly identical boundary signatures, discouraging observationally interchangeable decoder mechanisms.
Replace unconstrained output coordinates with a neural parameterization whose outputs are valid monotone profiles by construction, analogous to representing a Young diagram through nonnegative ordered row increments. Train the network against an explicit energy or negative log-probability while preserving the feasible geometry, rather than relying on penalties that permit invalid intermediate states.
Represent every predicted displacement and velocity as the sum of a prescribed boundary lift and a learned residual that is identically zero on the Dirichlet boundary. Feed the boundary velocity into the model through an explicit distributed-port feature and train an energy-balance residual so that the learned interior dynamics cannot inject arbitrary energy at the constrained boundary. This should eliminate boundary drift and reduce the burden on penalties or projection layers.
Insert a differentiable layer that enforces multiple affine consistency constraints by running several short strings of relaxed projections and averaging their outputs. Change the strings and weights across training steps, but impose bounded string length, positive averaging weights, and an almost-cyclic coverage rule so every constraint is revisited regularly. This creates an architecture-level analogue of dynamic string-averaging rather than applying one fixed projection order.
Augment a local convolutional block with a fractional nonlocal-gradient branch that aggregates directional feature differences over multiple spatial scales. The residual branch gives each location access to long-range variation while preserving the property that constant feature fields produce zero response. A learnable residual gate allows the network to suppress the branch if nonlocal interactions are unhelpful.
Represent an attention or routing state as a symmetric projector or fixed-spectrum positive semidefinite matrix and refine it using the paper's double-bracket flow instead of unconstrained gradient steps. The update rotates the state toward a task-derived Hermitian cost matrix while preserving its eigenvalues, so rank, trace, and spectral diversity remain fixed by construction.
Replace a recurrent sequence classifier's unconstrained hidden-state alarm head with an online truncated-signature state and a first-hitting-time linear detector. The module summarizes local order information and cross-channel interactions while preserving exact compositional updates, making it suitable for long streaming sequences and early-exit decisions.
Turn the paper's graphon refinement operation into an adaptive mixture-of-experts or relational-attention router whose number of blocks is learned rather than fixed. Split a block when its members produce incompatible constraint gradients, and merge blocks when their relation rows are statistically indistinguishable, yielding a compact architecture that grows only where the data require extra structure.
Replace isotropic particle noise or unconstrained pairwise graph updates by antisymmetric, relative-velocity-tangential noise. For each pair of particles, the update lies approximately in the hyperplane orthogonal to their relative displacement and has variance determined by a regularized soft-potential kernel. This should produce stochastic exploration while reducing center-of-mass drift and violations of kinetic-energy-like invariants.
Construct a sparse routing or graph-neural architecture whose activation gates satisfy a hard-core constraint: neighboring sites, experts, or token groups cannot be active simultaneously. Compare the same local routing rule on bipartite and random regular interaction graphs; the graph structure should change the maximum usable activation dimension and may also change optimization stability.
Replace fixed sequence-to-sequence attention with a dynamically maintained tree of connected token groups. Groups can be fused to reduce the number of attention units or split when their representation is heterogeneous, while hypergraph connectivity and nestedness ensure that every intermediate hierarchy remains valid.
Replace unconstrained MoE router logits with structured phase scores indexed by N-subsets of M ordered parameters. Each token is assigned to the dominant phase, while neighboring routing regions obey the Grassmannian rule that adjacent labels share N-1 indices, reducing arbitrary fragmented decision boundaries and encouraging smooth expert transitions.
Compute separate doubled-angle orientation order parameters for left and right image regions, then expose their sum and difference as symmetric and antisymmetric global features. This gives a network a low-dimensional inductive bias for global vertical structure versus left-right imbalance, while retaining magnitude channels that indicate when either readout is undefined because orientations cancel.
Replace a $K$-class softmax with $N$ binary hyperplane heads, where each class is represented by the signs of its projections onto fixed random directions. Train the embedding to reproduce these codewords and decode by nearest Hamming codeword. The paper's guarantee suggests that $N\approx 2\log_2 K+\log_2(1/\delta)$ can separate all class centers with high probability in sufficiently high dimension, giving a concrete width rule rather than choosing the number of binary heads heuristically.
Use a transport map \(Q_\theta\) from a fixed latent reference distribution to a data distribution, but expose only its locally averaged version \(\bar Q_{\theta,\sigma}(z)=\mathbb E_{u\sim K_\sigma(\cdot-z)}Q_\theta(u)\). Latent-space mollification integrates the pole-type influence singularity instead of allowing one training sample near \(Q_\theta(z)\) to dominate the quantile feature or its gradient.