# Bessel Totally-Positive Attention

- ID: 377
- Canonical URL: https://synthcore.org/idea/377/bessel-totally-positive-attention
- API JSON: https://synthcore.org/api/idea/377.json
- API Markdown: https://synthcore.org/api/idea/377.md
- Verification status: unverified
- Source: [arXiv:2607.02778](https://arxiv.org/abs/2607.02778)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: attention, transformer, regularization
- Math tags: linear-algebra, analysis, approximation-theory
- Ratings: usefulness 5/10; difficulty 6/10; novelty 8/10

## Idea description

Replace ordinary dot-product attention logits with a strictly totally positive kernel evaluated on positive, ordered scalar coordinates attached to queries and keys. Use the modified-Bessel kernel K(x,s)=I_s(x), whose every ordered minor is positive, then row-normalize it as an attention matrix. This creates an attention operator with a mathematically enforced anti-oscillatory structure rather than merely positive entries.

## Mathematical statement

The paper defines a kernel K:X×Y→R to be strictly totally positive of infinite order, written STP_infinity, when for every integer m≥1, every x_1<...<x_m in X, and every y_1<...<y_m in Y, det[K(x_i,y_j)]_{i,j=1}^m>0. Its concrete result is that the modified-Bessel kernel K(x,s)=I_s(x), for x>0 and s≥0, is STP_infinity. Here I_s(x) is the modified Bessel function of the first kind with real order s. For query coordinates x_i>0 and key coordinates s_j≥0 sorted increasingly, define W_{ij}=I_{s_j}(x_i). Every ordered square minor of W is positive. Row normalization A_{ij}=W_{ij}/sum_l W_{il} preserves the sign of every minor because it multiplies row i by a positive scalar. The proposed network exploits this determinant positivity as a structural bias toward non-oscillatory, order-preserving mixing.

## Key formulas

- $$\det\big[K(x_i,y_j)\big]_{i,j=1}^{m}>0\qquad\text{for every }m\ge1,\ x_1<\cdots<x_m,\ y_1<\cdots<y_m.$$
- $$K(x,s)=I_s(x),\qquad x>0,\quad s\ge0.$$
- $$W_{ij}=I_{s_j}(x_i),\qquad A_{ij}=\frac{W_{ij}}{\sum_{\ell=1}^{n}W_{i\ell}}.$$
- $$\mathcal{L}_{\mathrm{minor}}=\sum_{m=2}^{r}\frac{1}{|\mathcal{S}_m|}\sum_{(P,Q)\in\mathcal{S}_m}\operatorname{softplus}\!\left(\tau-\log\det\big[W_{p_aq_b}\big]_{a,b=1}^{m}\right).$$

## Implementation notes

(1) Integrate this into a small Transformer attention block. For each token, use a learned scalar query coordinate u_i and key coordinate v_j, transformed as x_i=softplus(u_i)+epsilon and s_j=softplus(v_j). To preserve the theorem's ordered-input condition, sort the coordinates within each sequence and apply the inverse permutation to the resulting attention output; alternatively use fixed normalized token positions for the first experiment. Compute W_{ij}=I_{s_j}(x_i) with a numerically stable special-function implementation, preferably through a log-Bessel routine, and form A by row normalization. Add the usual value projection AV, residual connection, and LayerNorm. (2) Pseudocode: x=softplus(q_coord)+eps; s=softplus(k_coord); order_q=argsort(x); order_k=argsort(s); x=x[order_q]; s=s[order_k]; logW=log_bessel_I(order=s[None,:], argument=x[:,None]); A=softmax(logW,dim=-1); y=unpermute_q(A @ V[order_k]). Use float64 for the first correctness test and clip x and s to a bounded interval such as [10^-3,10]. (3) The paper supplies the positivity guarantee; the implementation estimates no theorem constants. Empirically verify the guarantee by sampling ordered subsets and checking that determinants exceed 10^-8. For a cheaper baseline, retain standard attention and add the displayed low-order-minor penalty with m≤3, using random subsets (P,Q); compute log-determinants with slogdet and penalize nonpositive signs. (4) First experiment: train a 2-layer, 4-head model on sequential MNIST or a synthetic long-range copy/reversal task, comparing dot-product attention, Bessel attention, and Bessel attention plus the minor penalty at matched parameter count and FLOPs. Measure validation accuracy, gradient-norm variance, attention-matrix determinant sign violations, and performance after injecting alternating high-frequency noise into inputs. Success means fewer oscillatory attention patterns, lower gradient variance, and improved copy accuracy or robustness at equal training compute; a negative result is unchanged accuracy with substantially higher kernel-evaluation cost.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.