# Inverse-Square Fractional Attention

- ID: 647
- Canonical URL: https://synthcore.org/idea/647/inverse-square-fractional-attention
- API JSON: https://synthcore.org/api/idea/647.json
- API Markdown: https://synthcore.org/api/idea/647.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.09585](https://arxiv.org/abs/2607.09585)
- Category: architecture
- Solves: accuracy, sample-efficiency, scalability
- ML areas: attention, graph-nn, transformer
- Math tags: pde, functional-analysis, harmonic-analysis, geometry
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace or augment geometric attention on spatial or point-cloud tokens with a positive fractional kernel containing the paper's inverse-square origin factor. This gives tokens near a designated singular center a controlled increase in receptive-field influence while preserving a scale-invariant distance decay, which may help models represent cusp-like fields, radial singularities, and multiscale spatial interactions.

## Mathematical statement

The paper studies the inverse-square Schrödinger operator H_a=-Delta+a|x|^{-2} on R^d, with d>=3 and -(d-2)^2/4<=a<0. Define the Hardy singularity exponent sigma=(d-2-sqrt((d-2)^2+4a))/2, so 0<sigma<=(d-2)/2. For 0<s<d-2sigma, the positive kernel of the fractional inverse operator satisfies H_a^{-s/2}(x,y) approximately proportional to |x-y|^{s-d} times (min(|x|/|x-y|, |y|/|x-y|, 1))^{-sigma}, where x,y are points in R^d and min denotes the minimum of the three arguments. The factor |x-y|^{s-d} is fractional-integral decay, while the minimum factor amplifies interactions when either endpoint is close to the singular origin relative to their separation. The restriction s<d-2sigma keeps the origin singularity integrable. We use this positive kernel as a geometry-dependent attention prior while retaining content-dependent query-key logits.

## Key formulas

- $$H_a=-\Delta+\frac{a}{|x|^2},\qquad \sigma=\frac{d-2-\sqrt{(d-2)^2+4a}}{2},\qquad 0<s<d-2\sigma.$$
- $$H_a^{-s/2}(x,y)\simeq |x-y|^{s-d}\left(\frac{|x|}{|x-y|}\wedge\frac{|y|}{|x-y|}\wedge 1\right)^{-\sigma}.$$
- $$K_{ij}=\bigl(\|x_i-x_j\|_2+\varepsilon\bigr)^{s-d}\left(\frac{\|x_i-c\|_2+\varepsilon}{\|x_i-x_j\|_2+\varepsilon}\wedge\frac{\|x_j-c\|_2+\varepsilon}{\|x_i-x_j\|_2+\varepsilon}\wedge1\right)^{-\sigma}.$$
- $$A_{ij}=\frac{\exp\left((q_i^\top k_j)/\sqrt{h}+\lambda\log(K_{ij}+\varepsilon)\right)}{\sum_{\ell=1}^{N}\exp\left((q_i^\top k_\ell)/\sqrt{h}+\lambda\log(K_{i\ell}+\varepsilon)\right)},\qquad y_i=\sum_{j=1}^{N}A_{ij}v_j.$$

## Implementation notes

Integrate this at the attention-logit stage of a vision transformer, point-cloud transformer, or coordinate-aware graph neural network whose tokens have coordinates x_i in R^d. Let c be the singularity center; initially use the image center, point-cloud centroid, or a fixed physical origin. For each head, compute ordinary queries q_i, keys k_j, and values v_j. In parallel compute r_ij=||x_i-x_j||_2+epsilon, u_i=||x_i-c||_2+epsilon, and the kernel K_ij from the displayed formula. Add lambda log(K_ij+epsilon) to the standard scaled dot-product logits, mask invalid pairs, and apply softmax. Normalize coordinates to a bounded domain and use epsilon=10^-4. Initialize lambda=0 to isolate the effect, then learn a nonnegative lambda per head. Optionally learn sigma using sigma=(d-2)/2 sigmoid(b), and parameterize s=(d-2sigma) sigmoid(r) so the paper's admissibility condition is always satisfied. For large token sets, use the k nearest neighbors together with a few radial bins rather than dense pairwise attention. The paper determines the kernel form and admissible exponent range; c, lambda, and optional exponents are estimated by backpropagation. First test on a 32x32 synthetic coordinate-regression task with clipped singular targets such as f(x)=||x-c||^{-gamma}, followed by a point-cloud classification benchmark. Compare vanilla attention and relative-position bias at equal parameters and FLOPs. Success is lower error on singular or multiscale targets, faster loss descent, or better accuracy under token subsampling. Permuting coordinates is a negative control and should remove any gain.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built inverse-square fractional attention with the paper-derived positive kernel added to attention logits, plus numerical verification and a matched synthetic singular-field regression. The kernel was scale-homogeneous to relative error below 3e-8, matched the predicted 51.99x near-origin amplification, and passed the d=3 admissibility check. Across three fixed seeds, overall test MSE improved from mean 0.005827 to 0.005710 (~2.0%), but near-origin MSE slightly worsened (0.006125 to 0.006261), so the gain is modest and not specifically demonstrated as improved cusp accuracy.

### Mechanism check

- Verdict: Built inverse-square fractional attention with the paper-derived positive kernel added to attention logits, plus numerical verification and a matched synthetic singular-field regression. The kernel was scale-homogeneous to relative error below 3e-8, matched the predicted 51.99x near-origin amplification, and passed the d=3 admissibility check. Across three fixed seeds, overall test MSE improved from mean 0.005827 to 0.005710 (~2.0%), but near-origin MSE slightly worsened (0.006125 to 0.006261), so the gain is modest and not specifically demonstrated as improved cusp accuracy.
- Confidence: 7/10
- Limitations: Only a small synthetic 2D coordinate-token regression was tested; the kernel sanity check used d=3 while the toy attention model used 2D coordinates. No image or point-cloud benchmark, learned lambda or sigma, relative-position baseline, FLOP-normalized comparison, subsampling test, or permutation negative control was run.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0386
- Paired wins: 2/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/1426/bench_report.json)
- [bench_run.py](https://synthcore.org/code/1426/bench_run.py)
- [experiment.py](https://synthcore.org/code/1426/experiment.py)
- [repeat.py](https://synthcore.org/code/1426/repeat.py)
- [report.md](https://synthcore.org/code/1426/report.md)
- [report_bench_2026-09-04T211721.md](https://synthcore.org/code/1426/report_bench_2026-09-04T211721.md)
- [results.json](https://synthcore.org/code/1426/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1426)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.