Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.
Replace fixed sequence-to-sequence attention with a dynamically maintained tree of connected token groups. Groups can be fused to reduce the number of attention units or split when their representation is heterogeneous, while hypergraph connectivity and nestedness ensure that every intermediate hierarchy remains valid.
Replace unconstrained MoE router logits with structured phase scores indexed by N-subsets of M ordered parameters. Each token is assigned to the dominant phase, while neighboring routing regions obey the Grassmannian rule that adjacent labels share N-1 indices, reducing arbitrary fragmented decision boundaries and encouraging smooth expert transitions.
Replace unconstrained pairwise token grouping with a tree whose edges carry independent merge or cut variables. The connected components of the retained edges define a valid partition at every forward pass, while learned edge gates control the amount of token aggregation. A coarse component-level computation can then replace part of dense attention.
Compute separate doubled-angle orientation order parameters for left and right image regions, then expose their sum and difference as symmetric and antisymmetric global features. This gives a network a low-dimensional inductive bias for global vertical structure versus left-right imbalance, while retaining magnitude channels that indicate when either readout is undefined because orientations cancel.
Replace or augment a residual neural layer with a Fourier-domain scale-selective flow containing a learned second-order term and a fourth-order stabilizer. The block permits controlled low-frequency amplification, as required by the KS infrared mechanism, while damping high-frequency feature noise and preventing unbounded spectral growth.
Construct attention groups as hyperedges of a linear r-uniform hypergraph: every pair of tokens is allowed to share at most one group, while each token participates in approximately the same number of groups. Apply local attention inside each group and aggregate the outputs across groups. The construction inherits the paper's sharp capacity bound and prevents both redundant pair interactions and high-degree token hubs.
Replace a learned embedding for each integer with a compositional embedding of its variable-length Gray codeword. Add an auxiliary constraint that numerically adjacent values have nearby representations, while preserving the ordinary task loss so that the model can learn when numerical adjacency matters.
Replace a dense channel-mixing matrix by a sparse complex generalised weighing matrix W with exactly w nonzero entries in every row and column, then use U=W divided by square root of w as a norm-preserving mixer. Restricting to k=2 gives a real matrix with entries in {+1,-1}; k=4 supports signed phase rotations. The exact isometry should preserve signal and gradient norms while reducing channel-mixing cost.
Replace a dense token-mixing matrix in a sequence model with a fixed or learnable SBP derivative operator D=P^{-1}Q. The discrete integration-by-parts identity makes the interior mixing energy-neutral or boundary-dissipative, reducing exploding activations in deep residual stacks while preserving directional information along the sequence.
Add a profile-preserving exchange layer that mixes representations through pairwise state swaps, with weights shared over all positions. Unlike ordinary token-wise MLP mixing, the layer explicitly models interactions between two categorical states while respecting the symmetry of position permutations.
Add a geometric branch that converts ordered image contours into truncated signatures and feeds first-order displacement and second-order antisymmetric area features into the detector backbone. The area channel captures orientation and enclosed-region structure that ordinary edge magnitude or convolutional filters may miss, making the module suitable for thin cracks, scratches, bent boundaries, and small industrial defects.
Use the paper's crystal parenthesis matching as a deterministic local search over a sequence of MoE expert assignments. For adjacent experts i and i+1, pair assignments according to the crystal rule and move one unpaired i to i+1 only when this reduces expert-load imbalance. This provides a structured alternative to relying exclusively on an auxiliary load-balancing loss.
Replace activation-magnitude-based adaptive computation halting with a criterion based on the actual recurrent update and a local stability margin. The loop halts when the state change is small relative to state scale for several consecutive steps, avoiding pathological decisions when LayerNorm-driven dynamics cause the activation norm to collapse.
Decompose tensor-valued hidden states into invariant symmetric, alternating, and higher Young-symmetry channels before applying learned maps. This removes redundant tensor coordinates and prevents a neural layer from mixing incompatible representation types, yielding smaller equivariant modules with a cleaner inductive bias.
Add a coordinate-aware long-range aggregation branch whose singular low-frequency component is explicitly centered before it is mixed into token representations. The centering acts as a neural counterterm: constant or slowly varying value fields cannot accumulate an activation contribution that grows with context size, while local and higher-frequency interactions remain available through an ordinary attention residual branch.
Augment each token or graph node with a periodic latent position x_i and phase θ_i, then evolve these variables before attention or message passing. Tokens with similar phase attract in x, while tokens with similar position synchronize in θ, producing self-organized groups without an externally specified clustering objective. The coupling strengths J and K provide interpretable controls for aggregation and synchronization, and their sweep should expose the paper's four collective regimes and…
Replace or augment geometric attention on spatial or point-cloud tokens with a positive fractional kernel containing the paper's inverse-square origin factor. This gives tokens near a designated singular center a controlled increase in receptive-field influence while preserving a scale-invariant distance decay, which may help models represent cusp-like fields, radial singularities, and multiscale spatial interactions.
Use a symplectic Hamiltonian update as a recurrent or state-space neural block, preserving a learned modified energy across many layers or time steps. This targets residual and recurrent architectures where ordinary Euler updates accumulate drift during long rollouts.
Replace a fixed Fourier or spectral resolution in a neural operator or sequence model with a data-adaptive spectral cutoff. Keep only modes whose estimated signal energy exceeds the noise-amplification and discretization floor implied by the available number of trajectories and samples per trajectory. This should reduce overfitting to high-frequency sensor noise and preserve accuracy when the same model is deployed at different sampling resolutions.
Add a differentiable SNST layer before an EEG classifier or sequence model. For every local channel pair and wavelet band, compute the magnitude of the complex cross-channel analytic response, then average it over a controllable temporal window and concatenate it with ordinary channelwise features. This gives the model an explicit, phase-robust amplitude-coupling representation that is especially useful when labeled training data are scarce.
Represent a modular neural block as a word over primitive operators and identify operators that commute because they act on disjoint tensor supports. Canonicalize every equivalent word into one representative, allowing architecture search, caching, and parameter sharing to operate on equivalence classes rather than redundant layer orderings.
Replace a large graph-token set by a smaller set of connected wedge regions generated through adaptive two-seed shortest-path partitions. Each pooled token is the mean of the node features in its region, while the binary partition tree and region sizes are retained for unpooling or skip connections. This provides a deterministic, graph-aware alternative to arbitrary token merging that can be inserted before graph-transformer message passing.
Construct a sparse token-interaction graph from attention affinities and recursively split it by removing the closed neighborhoods of a small set of separator tokens. Separator tokens retain global communication, while the resulting anti-adjacent components perform local attention independently, reducing quadratic attention and KV-cache costs. The induced Erdos-Pósa theorem supplies a structural diagnostic: graphs with few anti-adjacent long-cycle or theta packings should admit small…
Replace an unconstrained feature vector of size n+1 by the coefficients of a homogeneous degree-n binary polynomial and make the layer transform through the irreducible symmetric-power representation of GL_2(R). For n=4 this creates a five-channel equivariant feature block whose transformation law is exact rather than learned through augmentation.