Every idea extracted from recent arXiv mathematics papers — verified and unverified. Click an idea to open its full card; badges show the empirical verdict.
Replace the usual random or elementwise-positive linear-attention feature map with a rank-one positive-semidefinite feature map derived from query and key vectors. For normalized inputs, the resulting kernel is the squared inner product, which is nonnegative and gives a geometrically structured interference pattern that is better suited to associative recall than an arbitrary low-rank feature map.
Split a recurrent or state-space model into a coarse quotient state \(z_t\) and a leaf or fibre state \(y_t\), where the quotient evolves autonomously and the fibre is driven conditionally by the quotient. Constrain the two transition operators to have independently measurable contraction or correlation rates, then allocate capacity and regularization to the slower branch. This is intended for sequence tasks containing both slowly evolving global variables and rapidly mixing local variables.
Treat every low-rank basis refresh as a change of coordinates instead of assuming that old optimizer coordinates remain aligned with the new basis. Transport the first moment with the basis-overlap matrix, but collapse the second moment to a rotation-blind isotropic estimate rather than applying the same coordinate transformation to elementwise squared moments. This should eliminate second-moment staleness while preserving the memory savings of low-rank optimization.
Construct a spatiotemporal neural block from localized functions of a learned parabolic operator instead of unrestricted attention or convolution. Use one filter for fine-scale diffusion and another for coarse-scale temporal aggregation, with the scale ratio controlling information propagation. The block should suppress distant interactions while still permitting long-range mixing through coarse filters.
Prune hidden units only after testing whether their nonlinear gate is task-visible and downstream-used. For ReLU, a unit is removable when its preactivation does not cross zero on the task patch or its outgoing weight column is zero; this is a more structural criterion than weight magnitude and can be applied during width search or post-training compression.
Replace nominal hidden-unit count by the number of distinct realized ReLU kink hyperplanes, then regularize or prune this effective count. Neurons whose normalized affine boundaries coincide can be exactly merged by summing their canonical coefficients, reducing memory without changing the represented function and aligning the model's complexity measure with the theorem.
Replace a discrete or one-hot recurrent state table with a low-dimensional vector memory whose event embeddings are orthogonal whenever the corresponding events are mutually exclusive in an input exclusivity graph. The module uses continuous state vectors and can therefore target dimension \(d=\xi(G)\), whereas a discrete state encoding is lower-bounded by \(N\geq\chi(G)\). This should be tested on graph-defined formal-language recognition tasks, where the graph is known and the claimed…