# Haar-Averaged Invariant Attention

- ID: 615
- Canonical URL: https://synthcore.org/idea/615/haar-averaged-invariant-attention
- API JSON: https://synthcore.org/api/idea/615.json
- API Markdown: https://synthcore.org/api/idea/615.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.08987](https://arxiv.org/abs/2607.08987)
- Category: architecture
- Solves: accuracy, stability, sample-efficiency
- ML areas: attention, transformer, data-augmentation
- Math tags: lie-groups, harmonic-analysis, geometry, probability
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Replace ordinary pairwise attention similarity by an affinity averaged over transformed keys or values. The resulting attention is invariant to the group action on either input and avoids requiring the network to learn identical attention patterns for every rotated or transformed copy.

## Mathematical statement

For a compact Lie group \(G\), let \(\mu_G\) denote normalized Haar measure, so \(\int_Gd\mu_G(g)=1\). Given a base similarity \(s(q,k)\), define the group-integrated invariant affinity \(\bar{s}(q,k)=\int_Gs(q,g\cdot k)\,d\mu_G(g)\). A practical Monte Carlo approximation uses \(g_1,\ldots,g_K\sim\mu_G\): \(\bar{s}_K(q,k)=K^{-1}\sum_{r=1}^Ks(q,g_r\cdot k)\). For Gaussian attention logits, use \(s(q,k)=-\|q-k\|^2/(2\tau^2)\), where \(\tau\) is a temperature, or exponentiate after averaging to obtain a positive kernel \(K_G(q,k)=K^{-1}\sum_r\exp[-\|q-g_r\cdot k\|^2/(2\tau^2)]\). The paper identifies integration over \(G\) as one of three invariant-kernel classes and proves that graph operators built from such kernels converge to operators on the quotient \(\mathcal{M}/G\). Thus the attention logits discard orbit-direction distinctions while retaining quotient geometry.

## Key formulas

- $$\bar{s}(q,k)=\int_G s(q,g\cdot k)\,d\mu_G(g)$$
- $$\bar{s}_K(q,k)=\frac{1}{K}\sum_{r=1}^{K}s(q,g_r\cdot k),\qquad g_r\sim\mu_G$$
- $$K_G(q,k)=\frac{1}{K}\sum_{r=1}^{K}\exp\left(-\frac{\|q-g_r\cdot k\|_2^2}{2\tau^2}\right)$$
- $$A_{ij}=\frac{K_G(q_i,k_j)}{\sum_{\ell}K_G(q_i,k_\ell)},\qquad \operatorname{Attn}(Q,K,V)_i=\sum_jA_{ij}v_j$$

## Implementation notes

1. Integration point: modify the attention-logit computation in one or more heads. Assume queries and keys contain a subvector on which the known group action is defined; for images this can be a feature map transformed by differentiable spatial rotation, while for point clouds it can be a rotated coordinate or steerable feature block. 2. Pseudocode: sample or deterministically select \(K\) group elements; for each query-key pair compute \(z_{ijr}=-(\|q_i-g_r\cdot k_j\|^2)/(2\tau^2)\); either use \(\ell_{ij}=\log\operatorname{meanexp}_r(z_{ijr})\) for numerically stable positive-kernel attention, or \(\ell_{ij}=K^{-1}\sum_rz_{ijr}\); apply row-wise softmax to \(\ell_{ij}\), then multiply by values. Cache transformed keys \(g_r\cdot k_j\) within a layer so the extra group dimension is computed once. 3. Haar integration and invariance are mathematical; Monte Carlo variance, \(K\), temperature \(\tau\), and whether values are transformed are engineering choices. Start with a fixed rotation grid, then test random Haar samples and antithetic pairs \((g,g^{-1})\). Use standard attention for heads not assigned symmetry-aware processing. 4. First cheap experiment: a 4-layer small ViT on rotated MNIST or CIFAR-10, with \(K=4,8,16\) SO(2) samples and the same parameter count as standard attention. Evaluate clean and unseen-angle accuracy, attention consistency between an image and its rotated copy, and training speed. A useful signal is substantially lower rotation-induced prediction variance and improved unseen-angle accuracy at equal augmentation budget; reject the method if the extra \(K\)-fold attention cost does not yield a better accuracy-per-FLOP tradeoff.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: inconclusive
- Verdict: Built and numerically verified Haar-averaged SO(2) Gaussian attention in verify_haar_attention.py. All three mechanism predictions manifested: dense Haar orbit variation was 2.3e-17, Monte Carlo error had observed slope -0.511 versus predicted -0.5, and orbit variation stayed at numerical precision across temperatures. In the toy attention test with keys rotated and queries fixed, output RMSE fell from 0.0869 for standard attention to 1.47e-13 for K=32 Haar attention; this confirms the claimed invariance mechanism, but not an accuracy or efficiency win.

### Mechanism check

- Verdict: Built and numerically verified Haar-averaged SO(2) Gaussian attention in verify_haar_attention.py. All three mechanism predictions manifested: dense Haar orbit variation was 2.3e-17, Monte Carlo error had observed slope -0.511 versus predicted -0.5, and orbit variation stayed at numerical precision across temperatures. In the toy attention test with keys rotated and queries fixed, output RMSE fell from 0.0869 for standard attention to 1.47e-13 for K=32 Haar attention; this confirms the claimed invariance mechanism, but not an accuracy or efficiency win.
- Confidence: 9/10
- Limitations: Only a NumPy toy kernel/attention experiment was tested; no neural-network training, image benchmark, learned values, random-Haar versus deterministic-grid comparison, GPU implementation, wall-clock/FLOP accounting, or accuracy-per-compute evaluation was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.32895
- Paired wins: 6/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_haar_attention.py](https://synthcore.org/code/276/bench_haar_attention.py)
- [bench_report.json](https://synthcore.org/code/276/bench_report.json)
- [report.md](https://synthcore.org/code/276/report.md)
- [report_bench_2026-08-31T000346.md](https://synthcore.org/code/276/report_bench_2026-08-31T000346.md)
- [results.json](https://synthcore.org/code/276/results.json)
- [verify_haar_attention.py](https://synthcore.org/code/276/verify_haar_attention.py)
- [Download all files as ZIP](https://synthcore.org/download/276)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.