# Weak-Bounded Riesz Attention

- ID: 2916
- Canonical URL: https://synthcore.org/idea/2916/weak-bounded-riesz-attention
- API JSON: https://synthcore.org/api/idea/2916.json
- API Markdown: https://synthcore.org/api/idea/2916.md
- Verification status: unverified
- Source: [arXiv:2608.31043](https://arxiv.org/abs/2608.31043)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: attention, cnn, regularization
- Math tags: harmonic-analysis, functional-analysis, measure-theory, geometry
- Ratings: usefulness 5/10; difficulty 5/10; novelty 7/10

## Idea description

Replace one local spatial aggregation in a CNN or vision transformer with a discretized Riesz potential whose kernel is proportional to $\|x-y\|^{-(n-s)}$. Normalize the layer using the paper's sharp weak-type constant and penalize empirical violations of the resulting tail bound, encouraging nonlocal context without allowing a small set of pixels or tokens to generate arbitrarily large responses.

## Mathematical statement

The paper defines the Riesz potential $I_s(f)(x)=\gamma_{n,s}\int_{\mathbb R^n}f(x-y)/\|y\|^{n-s}\,dy$, where $n$ is the spatial dimension, $s$ satisfies $0<s<\min\{n,2\}$, and $\gamma_{n,s}=2^{-s}\pi^{-n/2}\Gamma((n-s)/2)/\Gamma(s/2)$. Its sharp weak estimate is $\|I_s(f)\|_{L^{q,\infty}}\le C_{n,s}\|f\|_{L^1}$ with $q=n/(n-s)$ and $C_{n,s}=\gamma_{n,s}v_n^{(n-s)/n}\Gamma(s/2)\Gamma((n+2-s)/2)/\Gamma(n/2)$; $v_n$ is the volume of the unit ball in $\mathbb R^n$. The weak quasi-norm is $\|g\|_{L^{q,\infty}}=\sup_{t>0}t\,\mu(\{x:|g(x)|>t\})^{1/q}$, where $\mu$ is spatial Lebesgue measure. For a nonnegative feature map $f$, this implies the tail bound $\mu(\{I_s(f)>t\})\le (C_{n,s}\|f\|_1/t)^q$. In a discrete layer, replace integrals and measure by pixel/token sums and cell volume, estimate the input $L^1$ mass per channel, and use $C_{n,s}$ as a mathematically calibrated scale rather than an arbitrary attention temperature.

## Key formulas

- $$I_s(f)(x)=\gamma_{n,s}\int_{\mathbb{R}^n}\frac{f(x-y)}{\|y\|^{n-s}}\,dy,\qquad \gamma_{n,s}=2^{-s}\pi^{-\frac n2}\frac{\Gamma\!\left(\frac{n-s}{2}\right)}{\Gamma\!\left(\frac{s}{2}\right)}.$$
- $$\|I_s(f)\|_{L^{\frac{n}{n-s},\infty}}\leq\gamma_{n,s}v_n^{\frac{n-s}{n}}\frac{\Gamma(s/2)\Gamma((n+2-s)/2)}{\Gamma(n/2)}\|f\|_{L^1}.$$
- $$\|g\|_{L^{q,\infty}}=\sup_{t>0}t\,\mu\bigl(\{x:|g(x)|>t\}\bigr)^{1/q},\qquad q=\frac{n}{n-s},$$
- $$\mu\{x:I_s(f)(x)>t\}\leq\left(\frac{C_{n,s}\|f\|_{L^1}}{t}\right)^q,\qquad C_{n,s}=\gamma_{n,s}v_n^{\frac{n-s}{n}}\frac{\Gamma(s/2)\Gamma((n+2-s)/2)}{\Gamma(n/2)}.$$

## Implementation notes

(1) Integration point: use this as a replacement for one depthwise spatial-attention or large-kernel convolution block on an $H\times W$ feature map. Let $n=2$, choose $s\in(0,2)$, and apply the same scalar spatial kernel independently to each channel after a nonnegative transform such as $f=\operatorname{softplus}(x)$; a learned $1\times1$ projection can mix channels afterward. On a periodic grid, precompute the distance kernel $K_{ij}=\gamma_{2,s}(\|r_i-r_j\|+\varepsilon)^{-(2-s)}$, truncate it to radius $R$, and normalize the discrete sum by the cell area and kernel mass. (2) Pseudocode: `f=softplus(x); m=cell_area*sum(f, spatial); z=depthwise_conv_fft(f, K); q=n/(n-s); C=gamma(n,s)*v_n**((n-s)/n)*Gamma(s/2)*Gamma((n+2-s)/2)/Gamma(n/2); taus=quantile(z.detach().flatten(), grid); weak=max(taus * (cell_area*count(z>taus))**(1/q)); penalty=relu(weak-C*m)**2; y=proj(z)+x`. Add `lambda*penalty` to the training loss, or use `C*m` to initialize a final normalization scale. (3) Compute $\gamma_{n,s}$, $v_n=\pi^{n/2}/\Gamma(n/2+1)$, $q$, and $C_{n,s}$ exactly; estimate the weak quasi-norm using 16--32 empirical thresholds or sorted activation values. The theorem applies to the full untruncated continuous operator, so truncation, discretization, signed features, and learned kernel deviations must be tested empirically rather than treated as certified. (4) First experiment: train a small ResNet-18 and a ViT-Tiny on CIFAR-100 at $32\times32$, comparing a standard $7\times7$ depthwise convolution or local attention block against the Riesz block at $s=0.5,1.0,1.5$, with equal parameter count and matched FLOPs. Measure validation accuracy, maximum activation tail $\sup_t t\,\hat\mu(z>t)^{1/q}$, gradient norm variance, and robustness to impulse-corrupted pixels. Success means equal-or-better accuracy with fewer extreme activations, lower gradient spikes, and improved corruption accuracy; separately test whether truncation radius $R$ gives a favorable accuracy/FLOP tradeoff.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.