Machine-learning ideas tagged Generalization in the Solves taxonomy of the Math2NN corpus.
Replace dense attention over structured object tokens with attention over role-filler tensor-product representations. A learned query specifies both a role and a filler, retrieves objects matching that binding, extracts a target role, and rebinds the extracted filler into an output object.
Use an online estimate of the positive feedback gain among logits, routing probabilities, and representations to adjust the softmax temperature. Increase temperature when the estimated cyclic gain approaches the instability regime, preventing exponential amplification and router collapse without globally weakening all layers.
Replace ordinary row-softmax attention with a doubly stochastic Sinkhorn attention plan W, and periodically recover a gauge-fixed pairwise cost from W using the exact double-centering identity. Use this recovered cost to initialize or regularize a structured attention score, making the attention geometry identifiable despite arbitrary query and key row and column offsets.
Replace independent edge-type logits in a relational graph neural network with a mean-field fixed-point router derived from a colored ERGM. Each edge's color distribution is influenced by its own relation bias and by the expected number of rainbow triangles it forms with neighboring edges, allowing the model to learn coordinated multilayer structures.
Support conjunction queries over multiple roles without explicitly storing a huge tensor of repeated objects. Represent the required higher-order memory through query-dependent contractions, enabling compositional retrieval with memory that scales linearly in the number of objects.
Replace Euclidean momentum for selected neural parameters with a mirror or Bregman update, while using the paper's accelerated Newton direction for the objective step. Entropy geometry is especially suitable for softmax MoE routers, while Euclidean or log-barrier geometries can be used for unconstrained or positive parameters.
Replace an explicit attention step by an implicit Euler step that solves a proximal subproblem involving the Hopfield energy. The new state is evaluated inside the softmax self-consistently, which makes the method less sensitive to large step sizes and can prevent explicit attention from overshooting or tunneling between attraction basins.
Add a weighted reflection symmetry to an attention or graph-propagation matrix instead of requiring ordinary permutation equivariance. For paired positions or graph nodes related by an involution, penalize the failure of the propagation operator to commute with the weighted reflection; this makes all geometric multi-step propagations symmetry-compatible. The method is suitable for data with mirror, reversal, paired-agent, or left/right structure where the two sides have unequal importance…
Distill a teacher's attention into a student by matching sink mass and the normalized content distribution as separate targets rather than applying one KL divergence to the entire attention row. Use the Aitchison distance on the content composition, which compares relative token allocation and prevents a large common sink probability from overwhelming differences between content tokens.
Replace ordinary Wasserstein or arithmetic pooling of distribution-valued features with a barycenter whose individual quantile displacements are Huberized. Small changes between input distributions remain averaged quadratically, while a corrupted token, expert, graph neighborhood, or augmentation cannot move the pooled distribution arbitrarily far. The module is especially cheap for one-dimensional distributions represented by fixed quantile vectors.
Represent a graph, set, or attributed network as a measurable Z-valued kernel and train on lifted representatives while explicitly minimizing over node couplings. The quotient objective is invariant to relabeling by construction, while the lifted loss gives a dense correspondence signal that can stabilize graph attention and relational encoders.
Build a kernel aggregation layer whose output is a tangent vector field on the unit sphere and whose surface divergence is identically zero by construction. For each source point, use a matrix kernel obtained by applying a surface-rotated gradient in the query variable to a scalar zonal kernel; this is a differential-form version of the paper's matrix-valued construction. The layer can replace attention or message passing when the target dynamics are incompressible, such as spherical fluid…
Add a per-feature or per-token state that accumulates recent stimulation and decays when stimulation is absent, then use a nonlinear decreasing gain to suppress repeatedly activated features. This creates short-term adaptation without changing the core transformer or recurrent weights: familiar inputs are processed with reduced gain, while novel inputs recover their full response.
Use the relaxed QFT tensor-network topology as a trainable norm-preserving mixer inside a neural block, replacing a dense token-mixing matrix or an expensive global convolution. The network learns data-adapted global interactions while retaining structured O(N log^2 N) application and an exact cheap inverse, making it suitable for image tokens, long sequences, or reversible residual blocks.
Penalize short positive feedback cycles in an iterative neural module by suppressing products of absolute Jacobian blocks around the cycle. This targets the mechanism responsible for exponential temperature sensitivity rather than merely penalizing the total Jacobian norm, allowing strong feed-forward paths while controlling recurrent amplification.
Add a causal memory branch whose lag-response function is represented by a Bernstein polynomial with coefficients constrained to produce a nonnegative, decreasing, convex kernel. The branch aggregates past hidden states using this kernel, giving the model a learnable long-memory profile while preventing oscillatory, negative, or increasing historical influence.
Replace unconstrained token-mixing logits by a symmetric zero-row-sum response matrix generated from positive conductances on a small auxiliary electrical network. The resulting mixer has conservation and positivity structure, while circular minors have a prescribed sign pattern associated with positive grove measurements. This is especially suitable for graph neural networks and attention variants that need stable global diffusion rather than arbitrary dense affinities.
Replace an unconstrained nonlinearity on hyperbolic pairwise similarities with a function from the paper's exact Lorentz–Gram preserver family. The transformed similarity matrix remains realizable as Lorentz inner products of future-directed unit timelike vectors, allowing a network to sharpen or smooth hyperbolic neighborhoods without introducing geometrically impossible pairwise relations.
Replace an unconstrained attention or MoE routing matrix with a row-stochastic matrix that preserves a learned Gibbs-like distribution over groups. Anneal its temperature so that the router converges to a block-upper-triangular operator: mixing remains flexible within equal-energy groups, while cross-group traffic becomes directional rather than oscillating or collapsing. This creates a hierarchy-aware inductive bias and can reduce the number of active expert or attention connections.
Add a consistency loss for a neural model processing irregularly sampled functions or sequences, comparing its latent covariance representation before and after a time-coordinate warp. The target discrepancy should be near zero for affine transformations but should grow with the normalized curvature, bandwidth, and local sparsity of a nonlinear warp.
Replace an unconstrained positive higher-order gating function by a factorially normalized homogeneous polynomial whose coefficients are trained toward the Lorentzian Hessian signature. This gives multiplicative routing or attention interactions a controlled curvature pattern instead of allowing arbitrary unstable higher-order amplification.
Replace unconstrained momentum updates for selected neural-network parameter blocks by damped second-order motion on a unit sphere. The update learns directions while preserving the block norm exactly, and includes the centripetal term required for a valid acceleration constrained to the sphere rather than merely projecting gradients after an unconstrained step.
Use signed spanning-forest minor numerators to encourage a graph-structured neural layer to preserve independent multi-coordinate responses instead of collapsing several outputs onto the same direction. The determinant coefficients are only 0 or ±1, making the regularizer combinatorial and sign-exact rather than a noisy learned determinant surrogate.
Use a small set of known or trusted pairwise costs to estimate the effective entropic temperature of a Sinkhorn attention or mixture-of-experts routing layer directly from its observed transport plan. This provides a calibration controller that can detect over-concentrated routing and adjust epsilon without backpropagating through a costly temperature search.