# Fourier-compressed attention bias

- ID: 970
- Canonical URL: https://synthcore.org/idea/970/fourier-compressed-attention-bias
- API JSON: https://synthcore.org/api/idea/970.json
- API Markdown: https://synthcore.org/api/idea/970.md
- Verification status: unverified
- Source: [arXiv:2607.15598](https://arxiv.org/abs/2607.15598)
- Category: regularization
- Solves: memory, inference-speedup, generalization
- ML areas: attention, transformer, inference-speedup, regularization
- Math tags: harmonic-analysis, linear-algebra, graph-theory
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Replace an unconstrained relative-position attention-bias table by a learnable two-dimensional Fourier representation, or regularize the bias toward a small Fourier ratio. The ratio favors coefficient concentration without depending on the absolute scale of the bias, allowing the trained bias to be reconstructed from a small number of dominant frequencies.

## Mathematical statement

For a matrix-valued function \(f:\mathbb Z_N\times\mathbb Z_N\to\mathbb R\), the paper defines the normalized two-dimensional discrete Fourier transform \(\widehat f(m,n)=N^{-1}\sum_{x,y\in\mathbb Z_N}f(x,y)e^{-2\pi i(mx+ny)/N}\), where \(x,y,m,n\in\mathbb Z_N\). The Frobenius norm is \(\|M\|_2=(\sum_{i,j}|M_{ij}|^2)^{1/2}\), the entrywise norm is \(\|M\|_1=\sum_{i,j}|M_{ij}|\), and Parseval gives \(\|\widehat f\|_2=\|f\|_2\). The Fourier ratio is \(\operatorname{FR}(f)=\|\widehat f\|_1/\|\widehat f\|_2\); because \(\|z\|_1\geq\|z\|_2\), it is minimized by concentrated Fourier spectra. For a relative-position bias \(B\in\mathbb R^{H\times W}\), center and RMS-normalize it as \(\widetilde B=(B-\operatorname{mean}(B))/(\operatorname{RMS}(B)+\varepsilon)\), compute \(C=\operatorname{FFT2}(\widetilde B)\), and add \(\lambda(\|C\|_1/(\|C\|_2+\varepsilon)-1)\) to the task loss. At inference, retain the smallest frequency set \(S\) whose squared magnitudes capture energy fraction \(\rho\), then reconstruct \(B_S=\operatorname{IFFT2}(C\mathbf 1_S)\).

## Key formulas

- $$\widehat{f}(m,n)=\frac{1}{N}\sum_{x,y\in\mathbb Z_N}f(x,y)e^{-2\pi i(mx+ny)/N},\qquad \|\widehat f\|_2=\|f\|_2.$$
- $$\operatorname{FR}(f)=\frac{\|\widehat f\|_1}{\|\widehat f\|_2},\qquad \|M\|_1=\sum_{i,j}|M_{ij}|,\quad \|M\|_2=\left(\sum_{i,j}|M_{ij}|^2\right)^{1/2}.$$
- $$\mathcal L=\mathcal L_{\mathrm{task}}+\lambda\left(\frac{\|\operatorname{FFT2}(\widetilde B)\|_1}{\|\operatorname{FFT2}(\widetilde B)\|_2+\varepsilon}-1\right),\qquad \widetilde B=\frac{B-\operatorname{mean}(B)}{\operatorname{RMS}(B)+\varepsilon}.$$
- $$S=\min\left\{T:\sum_{(u,v)\in T}|C_{uv}|^2\geq\rho\sum_{u,v}|C_{uv}|^2\right\},\qquad B_S=\operatorname{IFFT2}\!\left(C\mathbf 1_S\right).$$

## Implementation notes

Integrate the method into the relative-position bias tensor of a vision transformer or windowed transformer. For each attention head \(h\), let \(B_h\in\mathbb R^{H\times W}\) be the learned bias indexed by relative vertical and horizontal offsets. Do not regularize the content-dependent \(QK^\top\) matrix in the first experiment, since its rows do not share one fixed spatial coordinate system. At each training step, compute \(\widetilde B_h=(B_h-\operatorname{mean}(B_h))/(\operatorname{RMS}(B_h)+\varepsilon)\), compute \(C_h=\operatorname{FFT2}(\widetilde B_h)\), and add \(R_h=\sum_{u,v}|C_{h,uv}|/(\sqrt{\sum_{u,v}|C_{h,uv}|^2}+\varepsilon)-1\) to the loss. Optimize \(\mathcal L_{task}+\lambda H^{-1}\sum_hR_h\), using \(\lambda=0\) during an initial 5--10% warmup and then linearly increasing it. After training, sort coefficients by \(|C_{h,uv}|^2\), retain the smallest set reaching energy fraction \(\rho\in\{0.90,0.95,0.99\}\), and reconstruct the bias with the inverse FFT. Store only the retained complex coefficients and their frequency indices, or parameterize the bias directly as a fixed Fourier dictionary with trainable amplitudes. The paper supplies the transform, Parseval normalization, and ratio; \(\lambda\), \(\varepsilon\), warmup, and \(\rho\) are empirical hyperparameters. First test DeiT-Tiny or a small ViT on CIFAR-100 or ImageNet-100 against an unrestricted bias table and an entrywise \(\ell_1\) penalty. Record validation accuracy, number of retained frequencies, bias-memory reduction, and inference latency. A successful result is accuracy within 0.2 percentage points of the unrestricted baseline with fewer than 10% of the bias coefficients, or higher accuracy at a fixed coefficient budget.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.