# RLCT-Certified Singular Attention

- ID: 942
- Canonical URL: https://synthcore.org/idea/942/rlct-certified-singular-attention
- API JSON: https://synthcore.org/api/idea/942.json
- API Markdown: https://synthcore.org/api/idea/942.md
- Verification status: unverified
- Source: [arXiv:2607.14991](https://arxiv.org/abs/2607.14991)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: attention, graph-nn, embedding
- Math tags: geometry, analysis, pde
- Ratings: usefulness 6/10; difficulty 5/10; novelty 9/10

## Idea description

Use an anisotropic singular relative-position kernel in attention or graph message passing, with its exponent constrained by the paper's local integrability threshold. The module can represent sharper directional interactions than an RBF while providing an explicit certificate that its spatial gradient belongs to a chosen L^p space.

## Mathematical statement

Let f:R^d -> R be a real-analytic function with f(0)=0, let kappa>0 be the singularity exponent, and define K(r)=|f(r)|^(-kappa), where r is a relative position. Near zero, the paper assumes |grad K(r)| is two-sided comparable to |grad f(r)| |f(r)|^(-(kappa+1)). For each divisor E_i in a real log-resolution of f and its Jacobian ideal, nu_i is the vanishing order of f, M_i is the vanishing order of the Jacobian ideal J_f=(partial_1 f,...,partial_d f), and a_i is the discrepancy exponent. The exact local integrability threshold is p*(kappa)=min_i (a_i+1)/((kappa+1)nu_i-M_i), and grad K belongs to local L^p exactly when p<p*(kappa). The real log-canonical threshold rlct_0(f) gives the sufficient bound pbar(kappa)=rlct_0(f)/(kappa+1). Thus choosing kappa < rlct_0(f)/p0 - 1 guarantees p*(kappa)>p0. For a quasi-homogeneous f with positive weights w_1,...,w_d, weighted degree q, and |w|=sum_i w_i, the paper gives p*(kappa)=|w|/(kappa q+max_i w_i).

## Key formulas

- $$\nabla K(r)\asymp |\nabla f(r)|\,|f(r)|^{-(\kappa+1)},\qquad K(r)=|f(r)|^{-\kappa}.$$
- $$\nabla K\in L^p_{\mathrm{loc}}\quad\Longleftrightarrow\quad p<p^*(\kappa):=\min_i\frac{a_i+1}{(\kappa+1)\nu_i-M_i}.$$
- $$\overline p(\kappa):=\frac{\mathrm{rlct}_0(f)}{\kappa+1}\leq p^*(\kappa),\qquad \kappa<\frac{\mathrm{rlct}_0(f)}{p_0}-1\ \Longrightarrow\ p^*(\kappa)>p_0.$$
- $$p^*(\kappa)=\frac{|w|}{\kappa q+\max_i w_i}\quad\text{for quasi-homogeneous }f.$$

## Implementation notes

Integrate the module into the relative-position bias of a point-cloud transformer, geometric transformer, or graph neural network. For positions x_i and values v_i, form r_ij=x_i-x_j and choose a fixed analytic polynomial such as f(r)=r_1^2+r_2^4+r_3^2. Use the regularized kernel k_ij=(abs(f(r_ij))+epsilon)^(-kappa), with epsilon positive during training. Either normalize it as a message-passing weight w_ij=k_ij/(sum_l k_il+delta), or add tau times log(k_ij+delta) to ordinary attention logits. Parameterize kappa=kappa_max sigmoid(theta), where kappa_max=rlct_0(f)/p0- margin; p0 is the desired gradient-integrability exponent and margin is a small positive safety constant.

Pseudocode:
```text
input positions x, values v, polynomial f, rlct, target p0
kappa_max = rlct / p0 - margin
kappa = kappa_max * sigmoid(raw_kappa)
for each batch:
    r[i,j] = x[i] - x[j]
    z[i,j] = abs(f(r[i,j])) + epsilon
    kernel[i,j] = z[i,j] ** (-kappa)
    weights[i,j] = kernel[i,j] / (sum_j kernel[i,j] + delta)
    output[i] = sum_j weights[i,j] * value_projection(v[j])
    optimize task loss
```

The mathematical certificate is computed once from the chosen f. For a quasi-homogeneous polynomial, calculate its weights and weighted degree and use the displayed closed form. For a quadratic isotropic f(r)=|r|^2 in dimension d, use kappa=(d-2)/2 and the paper's divisor data to recover p*=d/(d-1). Empirically estimate the safety margin by sampling relative positions at logarithmically spaced radii and fitting the slope of the sampled p-th moment of the spatial gradient. First test on ModelNet40 or a synthetic anisotropic interaction dataset with a small six-layer point-cloud transformer. Compare against RBF relative attention at matched FLOPs and parameters. Measure validation accuracy, loss descent at equal steps, gradient-norm quantiles, and robustness as epsilon decreases. Success requires equal-or-better accuracy and no heavy-tailed gradient explosion.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.