Every idea extracted from recent arXiv mathematics papers — verified and unverified. Click an idea to open its full card; badges show the empirical verdict.
Replace dense attention over structured object tokens with attention over role-filler tensor-product representations. A learned query specifies both a role and a filler, retrieves objects matching that binding, extracts a target role, and rebinds the extracted filler into an output object.
Replace ordinary modality-specific residual fusion with a switched observer whose latent correction depends on the currently available channel. The individual channels are allowed to be insufficient to reconstruct the latent state; stability is enforced over the full switching cycle, so complementary intermittent observations can jointly maintain a stable representation.
Replace a standard softmax-gradient update for a probability vector with a two-stage KL Mirror-Prox update. The predictor evaluates the population-dependent cost at the current distribution, and the corrector evaluates it at the predicted distribution, reducing oscillation when routing or attention costs are coupled across tokens or samples.
Learn a low-dimensional Koopman operator from successive states of an iterative neural system, such as debate agents, recurrent refinement blocks, or diffusion denoising trajectories. Use the magnitude of the subdominant eigenvalue to predict the remaining number of rounds required for disagreement to fall below a target tolerance, and stop computation when the predicted deadline is reached rather than using a fixed round budget.
Replace consecutive or randomly assigned transformed KV coefficients with groups whose variance-volume is approximately equal. Train one equal-size vector-quantizer codebook per group, so a fixed-width cache does not waste its low-rate budget by forcing high-variance and low-variance coordinates into badly mismatched groups. This is a drop-in quantization-layout change that can be applied to keys, values, or both.
Use an online estimate of the positive feedback gain among logits, routing probabilities, and representations to adjust the softmax temperature. Increase temperature when the estimated cyclic gain approaches the instability regime, preventing exponential amplification and router collapse without globally weakening all layers.
Replace the usual random or elementwise-positive linear-attention feature map with a rank-one positive-semidefinite feature map derived from query and key vectors. For normalized inputs, the resulting kernel is the squared inner product, which is nonnegative and gives a geometrically structured interference pattern that is better suited to associative recall than an arbitrary low-rank feature map.
Replace ordinary row-softmax attention with a doubly stochastic Sinkhorn attention plan W, and periodically recover a gauge-fixed pairwise cost from W using the exact double-centering identity. Use this recovered cost to initialize or regularize a structured attention score, making the attention geometry identifiable despite arbitrary query and key row and column offsets.
Augment a 1D neural operator or transformer with explicit tokens for detected discontinuities. Advance each front analytically using the local Rankine–Hugoniot speed and train the network only to reconstruct smooth regions and the residual caused by source terms and grid resolution.
Replace an explicit attention step by an implicit Euler step that solves a proximal subproblem involving the Hopfield energy. The new state is evaluated inside the softmax self-consistently, which makes the method less sensitive to large step sizes and can prevent explicit attention from overshooting or tunneling between attraction basins.
Insert a linear Johnson–Lindenstrauss bottleneck around a set of jointly processed representations, choosing its width from the sharp finite-set dimension bound rather than from the model's nominal hidden size. The projection should preserve pairwise distances between tokens, patches, or retrieved items, allowing a downstream attention or MLP block to operate at lower width while retaining the geometry relevant to similarity computations.
Replace a wide collection of interchangeable near-zero branches with a module whose output is explicitly a quadratic form in the branch-weight Gram matrix. The module preserves the paper's leading-order behavior while making the relevant collective variable explicit and allowing low-rank parameterizations.
Parameterize an entropic OT cost only in directions that can change the transport plan, removing row-plus-column potential directions that are invisible because of OT gauge invariance. Whiten the remaining feature coordinates using their empirical covariance, producing an OT layer whose identifiable parameters have substantially more uniform sensitivity.
Use predicted covariance reduction as a differentiable gate for selecting tokens, views, sensors, or retrieved demonstrations. The gate favors inputs with high expected information gain while accounting for acquisition cost, turning attention and data collection into active observability optimization.
Construct overlapping attention windows from a block design instead of using one dense sequence-by-sequence attention matrix. Every token appears in exactly $r$ windows and every token pair co-occurs in exactly $\lambda$ windows, giving uniform coverage and avoiding the uneven connectivity of arbitrary sparse masks.
Compress a transformer KV cache by selecting actual past tokens whose key or hidden-state columns form a stable basis for all cached tokens. Instead of retaining tokens with the largest attention scores or leverage scores independently, compute rank-revealing pivoting of the leading right-singular-vector matrix and retain its pivot columns, then evaluate attention using the representatives plus an optional low-cost residual correction.
Use a low-rank Tucker reconstruction as a structured backbone and quantize only its residual after an orthogonal rotation. The rotation preserves residual energy but redistributes it across coordinates, reducing dynamic-range imbalance and making 2- or 4-bit uniform quantization less damaging than direct quantization of the original KV tensor.
Add a causal memory branch whose lag-response function is represented by a Bernstein polynomial with coefficients constrained to produce a nonnegative, decreasing, convex kernel. The branch aggregates past hidden states using this kernel, giving the model a learnable long-memory profile while preventing oscillatory, negative, or increasing historical influence.
Partition tokens into a rectangular grid and use Latin-square labels to define several sparse attention heads. Each head attends only to tokens sharing a row, column, or Latin label, while orthogonality guarantees that every pair of labels occurs at most once, reducing systematic blind spots and repeated collisions. The resulting masks are deterministic, reusable across examples, and can be generated without learned routing scores.
Replace ordinary pairwise attention similarity by an affinity averaged over transformed keys or values. The resulting attention is invariant to the group action on either input and avoids requiring the network to learn identical attention patterns for every rotated or transformed copy.
Replace quadratic self-attention over a sequence with a bank of K auxiliary exponentially decaying states whose rates are fitted directly from the empirical autocorrelation of the sequence features. Each mode captures a distinct time scale, so the module can represent short- and long-range dependencies with O(TK) computation and O(K) recurrent memory rather than storing all previous tokens. Constrain decay rates to be positive and use the paper's extended Markovian block structure to obtain a…
Replace dense coarse-to-fine cross-attention at multiresolution interfaces with a sparse, nonnegative overlap operator whose weighted feature average is exactly conserved between the two resolutions. Use this operator as a low-order path and blend it with an unrestricted neural cross-attention path through a convex limiter that keeps features inside a box or simplex domain. The construction is especially suitable for adaptive token grids, hierarchical graph neural networks, neural operators…
Use a smoothed Burg entropy as the mirror map in a proximal-gradient optimizer for positive or simplex-valued neural parameters. The optimizer performs a Bregman-proximal step instead of an additive Euclidean update, while the smoothing parameter avoids the singularity of ordinary Burg entropy at zero.
Construct a spatiotemporal neural block from localized functions of a learned parabolic operator instead of unrestricted attention or convolution. Use one filter for fine-scale diffusion and another for coarse-scale temporal aggregation, with the scale ratio controlling information propagation. The block should suppress distant interactions while still permitting long-range mixing through coarse filters.