# Ground-State Fractional Attention

- ID: 740
- Canonical URL: https://synthcore.org/idea/740/ground-state-fractional-attention
- API JSON: https://synthcore.org/api/idea/740.json
- API Markdown: https://synthcore.org/api/idea/740.md
- Verification status: unverified
- Source: [arXiv:2607.11280](https://arxiv.org/abs/2607.11280)
- Category: architecture
- Solves: accuracy, stability, scalability
- ML areas: attention, transformer, graph-nn
- Math tags: harmonic-analysis, functional-analysis, pde, geometry
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Replace or augment relative-position attention with a positive fractional-integration mixing kernel whose radial behavior has separate inner and outer power laws. Tokens close to one another interact through the usual fractional singularity, while tokens near different radial scales receive a ground-state correction that can improve multiscale information transport without introducing a dense learned positional table.

## Mathematical statement

The paper's clean-kernel estimate states that for the fractional power H^{-s/2} of a radial Schrödinger operator, its kernel satisfies K_s^H(x,y) \simeq |x-y|^{s-d}[U(|x|)U(|y|)]/[U(|x|+|x-y|)U(|y|+|x-y|)]. Here x,y\in\mathbb{R}^d are token coordinates, d is their coordinate dimension, s is the fractional order, and U is a positive harmonic ground state. The two asymptotic regimes are U(r)\simeq r^{-\sigma_0} for 0<r\leq 1 and U(r)\simeq r^{-\sigma_\infty} for r\geq 1, with -d/2<\sigma_0,\sigma_\infty<d/2. The clean formula is asserted in the maximal range 0<s<\min\{d,d-2\sigma_0,d-2\sigma_\infty\}. We use the exact positive kernel as an attention bias or fixed mixing matrix; positivity and the explicit endpoint-dependent correction are the transferable assets. For numerical stability define \rho_{ij}=\sqrt{\|x_i-x_j\|_2^2+\varepsilon^2} and a continuous broken-power ground state U(r)=(r+\varepsilon)^{-\sigma_0} for r\leq 1 and U(r)=C(r+\varepsilon)^{-\sigma_\infty} for r>1, where C approximately matches the two branches at r=1.

## Key formulas

- $$K_s^H(x,y)\simeq |x-y|^{s-d}\frac{U(|x|)U(|y|)}{U(|x|+|x-y|)U(|y|+|x-y|)}.$$
- $$U(r)\simeq r^{-\sigma_0}\quad(0<r\leq 1),\qquad U(r)\simeq r^{-\sigma_\infty}\quad(r\geq 1).$$
- $$0<s<\min\{d,d-2\sigma_0,d-2\sigma_\infty\}.$$
- $$A_{ij}=\frac{\exp\!\left(\log K_{ij}/\tau+b_{ij}\right)}{\sum_{k=1}^{n}\exp\!\left(\log K_{ik}/\tau+b_{ik}\right)},\qquad Y_i=X_i+\gamma\sum_{j=1}^{n}A_{ij}V_j.$$

## Implementation notes

(1) Integration point: insert the module after the value projection in one or two transformer blocks. Inputs are token features X\in\mathbb{R}^{n    imes c}, coordinates x_i\in\mathbb{R}^d, and projected values V=XW_V. Use the geometric kernel as an additive attention logit or as a separate residual mixer; initially keep the kernel fixed. (2) Pseudocode: choose s, sigma0, sigma_inf, epsilon, and temperature tau. For every pair i,j compute r_i=||x_i||, r_j=||x_j||, delta_ij=sqrt(||x_i-x_j||^2+epsilon^2), define U(r)=(r+epsilon)^(-sigma0) when r<=1 and U(r)=C(r+epsilon)^(-sigma_inf) otherwise, then calculate logK_ij=(s-d)log(delta_ij)+logU(r_i)+logU(r_j)-logU(r_i+delta_ij)-logU(r_j+delta_ij). Form A with a row-wise softmax of logK/tau, optionally add the learned content logit q_i^T k_j/sqrt(c_h), and return X+gamma A V. Restrict s to the admissible interval from the paper and use a k-nearest-neighbor mask for large n. (3) Computed from the mathematics: the pairwise power kernel, the two-regime U, and the admissible s range. Estimated empirically: tau, residual scale gamma, sigma values if made learnable, and neighbor count k. Monitor row entropy because the singular distance term can collapse attention onto self-neighbors; exclude self edges or cap logK when necessary. (4) First experiment: train a ViT-Tiny or 6-layer transformer on CIFAR-100 or ImageNet-100 with normalized 2D patch coordinates centered at the image midpoint. Compare standard relative-position attention, fixed fractional attention with sigma0=sigma_inf=0, and the broken-power version at matched FLOPs. Evaluate clean accuracy, shifted/cropped-object robustness, and feature-rescaling stress tests. Success is higher validation or corruption accuracy at equal training cost, smoother gradient norms, and no attention-entropy collapse. Also compare full and k-neighbor sparse variants and record attention-memory reduction at matched accuracy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.