Unverified 2026

Bessel Totally-Positive Attention

Implementation & benchmark of arXiv:2607.02778 — Strict Total Positivity from Spectral Darboux and Toeplitz Smoothing Mechanisms

Usefulness5/10

Difficulty6/10

Novelty8/10

Architecture Stability Accuracy Generalization Attention Transformer Regularization Linear algebra Analysis Approximation theory

Source paper: Strict Total Positivity from Spectral Darboux and Toeplitz Smoothing Mechanisms arXiv:2607.02778 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace ordinary dot-product attention logits with a strictly totally positive kernel evaluated on positive, ordered scalar coordinates attached to queries and keys. Use the modified-Bessel kernel K(x,s)=I_s(x), whose every ordered minor is positive, then row-normalize it as an attention matrix. This creates an attention operator with a mathematically enforced anti-oscillatory structure rather than merely positive entries.

Formulas

$$\det\big[K(x_i,y_j)\big]_{i,j=1}^{m}>0\qquad\text{for every }m\ge1,\ x_1<\cdots<x_m,\ y_1<\cdots<y_m.$$

$$K(x,s)=I_s(x),\qquad x>0,\quad s\ge0.$$

$$W_{ij}=I_{s_j}(x_i),\qquad A_{ij}=\frac{W_{ij}}{\sum_{\ell=1}^{n}W_{i\ell}}.$$

$$\mathcal{L}_{\mathrm{minor}}=\sum_{m=2}^{r}\frac{1}{|\mathcal{S}_m|}\sum_{(P,Q)\in\mathcal{S}_m}\operatorname{softplus}\!\left(\tau-\log\det\big[W_{p_aq_b}\big]_{a,b=1}^{m}\right).$$

Mathematical statement

The paper defines a kernel K:X×Y→R to be strictly totally positive of infinite order, written STP_infinity, when for every integer m≥1, every x_1<...<x_m in X, and every y_1<...<y_m in Y, det[K(x_i,y_j)]_{i,j=1}^m>0. Its concrete result is that the modified-Bessel kernel K(x,s)=I_s(x), for x>0 and s≥0, is STP_infinity. Here I_s(x) is the modified Bessel function of the first kind with real order s. For query coordinates x_i>0 and key coordinates s_j≥0 sorted increasingly, define W_{ij}=I_{s_j}(x_i). Every ordered square minor of W is positive. Row normalization A_{ij}=W_{ij}/sum_l W_{il} preserves the sign of every minor because it multiplies row i by a positive scalar. The proposed network exploits this determinant positivity as a structural bias toward non-oscillatory, order-preserving mixing.

Implementation notes

(1) Integrate this into a small Transformer attention block. For each token, use a learned scalar query coordinate u_i and key coordinate v_j, transformed as x_i=softplus(u_i)+epsilon and s_j=softplus(v_j). To preserve the theorem's ordered-input condition, sort the coordinates within each sequence and apply the inverse permutation to the resulting attention output; alternatively use fixed normalized token positions for the first experiment. Compute W_{ij}=I_{s_j}(x_i) with a numerically stable special-function implementation, preferably through a log-Bessel routine, and form A by row normalization. Add the usual value projection AV, residual connection, and LayerNorm. (2) Pseudocode: x=softplus(q_coord)+eps; s=softplus(k_coord); order_q=argsort(x); order_k=argsort(s); x=x[order_q]; s=s[order_k]; logW=log_bessel_I(order=s[None,:], argument=x[:,None]); A=softmax(logW,dim=-1); y=unpermute_q(A @ V[order_k]). Use float64 for the first correctness test and clip x and s to a bounded interval such as [10^-3,10]. (3) The paper supplies the positivity guarantee; the implementation estimates no theorem constants. Empirically verify the guarantee by sampling ordered subsets and checking that determinants exceed 10^-8. For a cheaper baseline, retain standard attention and add the displayed low-order-minor penalty with m≤3, using random subsets (P,Q); compute log-determinants with slogdet and penalize nonpositive signs. (4) First experiment: train a 2-layer, 4-head model on sequential MNIST or a synthetic long-range copy/reversal task, comparing dot-product attention, Bessel attention, and Bessel attention plus the minor penalty at matched parameter count and FLOPs. Measure validation accuracy, gradient-norm variance, attention-matrix determinant sign violations, and performance after injecting alternating high-frequency noise into inputs. Success means fewer oscillatory attention patterns, lower gradient variance, and improved copy accuracy or robustness at equal training compute; a negative result is unchanged accuracy with substantially higher kernel-evaluation cost.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.