Ground-State Fractional Attention

(1) Integration point: insert the module after the value projection in one or two transformer blocks. Inputs are token features X\in\mathbb{R}^{n imes c}, coordinates x_i\in\mathbb{R}^d, and projected values V=XW_V. Use the geometric kernel as an additive attention logit or as a separate residual mixer; initially keep the kernel fixed. (2) Pseudocode: choose s, sigma0, sigma_inf, epsilon, and temperature tau. For every pair i,j compute r_i=||x_i||, r_j=||x_j||, delta_ij=sqrt(||x_i-x_j||^2+epsilon^2), define U(r)=(r+epsilon)^(-sigma0) when r<=1 and U(r)=C(r+epsilon)^(-sigma_inf) otherwise, then calculate logK_ij=(s-d)log(delta_ij)+logU(r_i)+logU(r_j)-logU(r_i+delta_ij)-logU(r_j+delta_ij). Form A with a row-wise softmax of logK/tau, optionally add the learned content logit q_i^T k_j/sqrt(c_h), and return X+gamma A V. Restrict s to the admissible interval from the paper and use a k-nearest-neighbor mask for large n. (3) Computed from the mathematics: the pairwise power kernel, the two-regime U, and the admissible s range. Estimated empirically: tau, residual scale gamma, sigma values if made learnable, and neighbor count k. Monitor row entropy because the singular distance term can collapse attention onto self-neighbors; exclude self edges or cap logK when necessary. (4) First experiment: train a ViT-Tiny or 6-layer transformer on CIFAR-100 or ImageNet-100 with normalized 2D patch coordinates centered at the image midpoint. Compare standard relative-position attention, fixed fractional attention with sigma0=sigma_inf=0, and the broken-power version at matched FLOPs. Evaluate clean accuracy, shifted/cropped-object robustness, and feature-rescaling stress tests. Success is higher validation or corruption accuracy at equal training cost, smoother gradient norms, and no attention-entropy collapse. Also compare full and k-neighbor sparse variants and record attention-memory reduction at matched accuracy.