Attention variants, state-space and recurrent cells, normalization and token-mixing schemes — each tested against the standard block it replaces.
Replace a free-form order-dependent gate with a positive mixture of Mellin powers $(1+s)^{-a}$. This gives a small, interpretable module whose response across the order variable is automatically generated by a positive measure and therefore inherits complete monotonicity, log-convexity, and Hankel-moment structure.
Replace a flat stochastic MoE router by a two-level continuous-time routing model: experts within a group communicate rapidly, while transitions between groups occur slowly. Use the effective class-level stationary distribution as a soft load-balancing prior, reducing routing oscillation while preserving expert specialization.
Parameterize a relative-position or lag-decay function as a finite positive mixture of exponentials instead of learning arbitrary attention bias values. The resulting kernel is completely monotone on positive distances, so it is nonnegative, decreasing, and has alternating derivative signs; the mixture provides several learned memory scales without allowing oscillatory or unstable long-range biases.
Store a convex object as a direction-indexed vertex tuple and implement composition of objects through componentwise Minkowski addition and nonnegative scaling. This creates a structured residual or compositional layer where convexification is nonexpansive, making perturbation amplification controllable and avoiding repeated generic geometric optimization.
Replace random Fourier features or a dense sinusoidal positional encoding with a compact bank whose frequencies are the continued-fraction denominators of an irrational number. Inverse-frequency amplitudes provide multiscale structure with a controlled sub-Lipschitz regularity profile, while lacunarity reduces the number of frequencies needed to represent oscillatory structure.
Replace an unconstrained high-order polynomial interaction module with features generated by Gaussian matrix contractions and their exact Wick expansion. The resulting interactions are sums of products of power-sum invariants, with coefficients fixed by perfect-matching counts, providing a low-parameter inductive bias for permutation- or orthogonal-structured data.
Augment a mesh or graph neural network with an explicit low-dimensional channel for topological circulation or flux modes. The network predicts a local gauge-fixed field u and global coefficients a, then reconstructs the physical field as y = u + Ha, so local message passing does not need to synthesize global modes through many layers.
Replace raw braid-generator sequences by sequences of positive simple Garside factors obtained from the left-greedy normal form. Because powers of \(\Delta\) lie in the Hilden subgroup, they can be removed while preserving the relevant double-coset presentation, reducing non-uniqueness and often shortening the sequence. Feed the resulting factor tokens to a Transformer or sequence classifier, and train it to be invariant to inserted removable \(\Delta\)-powers.
Construct a mesh neural network with node, edge, face, and cell feature spaces modeled on the four spaces of the discrete elasticity complex. Replace unconstrained cross-order message passing by fixed incidence and geometric operators whose compositions vanish exactly, so gradient-like, incompatibility-like, and divergence-like features cannot contain algebraically spurious components.
Parameterize a multi-channel two-dimensional convolutional operator through a learned filter bank B, then use the composed operator B*B as the layer response. Its Fourier response is positive semidefinite exactly at every spatial frequency, enabling stable smoothing or diffusion-like residual updates without frequency-grid penalty terms.
Replace one deterministic residual update with a short cyclic composition of learned vector fields evaluated for randomized, short run times. Because finite compositions of noncommuting flows generate directional-derivative and Lie-bracket terms, changing the cycle order gives the network an explicit, low-cost way to learn drift directions that are unavailable from the individual vector fields alone.
Replace a conventional scalar activation by a geometrically indexed family of affine pieces whose slope changes with the logarithmic magnitude of the input. The same two endpoint parameters are reused across all scales, giving a compact, explicitly scale-aware activation that can represent different responses for exponentially separated activation magnitudes.
Replace Fourier or sinusoidal one-dimensional coordinate features with a trainable radical layer f(x)=sum_i c_i sqrt(P_i(x)), where every P_i is a strictly positive quadratic. A nonzero scalar output formed by such a layer has at most 2n distinct real zeros, providing an explicit bound on sign changes and suppressing uncontrolled ringing. Use the radical features as an input embedding for a conventional MLP or neural implicit field.
Prune higher-order tensor or polynomial interactions while preserving an exchange-closed support instead of independently retaining the largest weights. The resulting sparse interaction pattern avoids arbitrary holes and retains structured substitutions between coordinates, potentially improving parameter efficiency and robustness after pruning.
Treat a finite shallow network as a discrete signed measure over ridge atoms and grow or prune neurons according to their contribution to total variation. This turns width selection into an atomic approximation procedure: add neurons correlated with the current residual and remove coefficients that consume budget without contributing materially.
Constrain a channel-mixing layer to be a product of nonnegative bidiagonal matrices, rather than an unconstrained dense matrix. The resulting totally nonnegative operator is predicted not to increase sign oscillations in ordered channel features, potentially reducing high-frequency feature noise and making deep stacks more stable.
Replace one local spatial aggregation in a CNN or vision transformer with a discretized Riesz potential whose kernel is proportional to $\|x-y\|^{-(n-s)}$. Normalize the layer using the paper's sharp weak-type constant and penalize empirical violations of the resulting tail bound, encouraging nonlocal context without allowing a small set of pixels or tokens to generate arbitrarily large responses.
Approximate a graph's adjacency by a learned abelian Cayley host and use one shared message-passing operator for every edge in the same inverse-pair generator class. Keep only the unexplained original edges as a residual branch, so the layer interpolates between a parameter-efficient group convolution and ordinary graph message passing.
Replace raw powers or unconstrained polynomial spectral features with normalized Jacobi features whose amplitude is provably bounded on the entire input interval. Use trainable mixtures of these features in a positional encoding, graph spectral layer, or MLP front end, while preserving the theorem's normalization and optionally constraining the learned mixture norm.
Replace raw polynomial features in a scalar MLP expansion with endpoint-weighted orthonormal Jacobi features. The paper's envelope gives a degree- and parameter-aware scale for each feature, preventing high-degree terms or endpoint behavior from dominating gradients while preserving a richer approximation basis than low-degree monomials.
Replace a globally shared latent transformation group by a source-dependent collection of valid transformation paths. A feature at latent point z is transported only along paths whose transformed coordinate never reaches the singular locus, while homotopic paths are identified and composable paths are concatenated. This should let an equivariant model represent branched or incomplete symmetries that ordinary group-equivariant layers must discard.
Replace ordinary simplicial incidence matrices in a graph or mesh neural network by integer-ratio weighted incidences derived from a divisibility hierarchy on simplex weights. The resulting up/down message-passing operators preserve exact chain cancellation, so features propagated around a filled simplex cannot create spurious boundary signals. Train the weights either from known metadata or as positive integer powers of a small prime, while retaining an ordinary-incidence baseline for ablation.
Parameterize a learned token metric as a nonnegative sum of sparse integral rank-one projections with unimodular support, rather than learning an unconstrained dense positive-semidefinite matrix. Graph-incidence covectors give an immediately implementable support family, while nonnegative coefficients guarantee positive semidefiniteness by construction.
Use reverse plane partitions of a minuscule heap as the discrete codebook for a VQ-VAE or discrete sequence model. Codes are not arbitrary indices: each code is an order-preserving array, and crystal raising/lowering operators define a sparse, semantically structured neighborhood graph for augmentation, routing, and metric regularization.