✗ Failed on benchmark 2026

Haar-Averaged Invariant Attention

Implementation & benchmark of arXiv:2607.08987 — Group Invariant Spectral Embedding

Usefulness7/10

Difficulty5/10

Novelty5/10

Architecture Accuracy Stability Sample efficiency Attention Transformer Data augmentation Lie groups Harmonic analysis Geometry Probability

Source paper: Group Invariant Spectral Embedding arXiv:2607.08987 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace ordinary pairwise attention similarity by an affinity averaged over transformed keys or values. The resulting attention is invariant to the group action on either input and avoids requiring the network to learn identical attention patterns for every rotated or transformed copy.

Formulas

$$\bar{s}(q,k)=\int_G s(q,g\cdot k)\,d\mu_G(g)$$

$$\bar{s}_K(q,k)=\frac{1}{K}\sum_{r=1}^{K}s(q,g_r\cdot k),\qquad g_r\sim\mu_G$$

$$K_G(q,k)=\frac{1}{K}\sum_{r=1}^{K}\exp\left(-\frac{\|q-g_r\cdot k\|_2^2}{2\tau^2}\right)$$

$$A_{ij}=\frac{K_G(q_i,k_j)}{\sum_{\ell}K_G(q_i,k_\ell)},\qquad \operatorname{Attn}(Q,K,V)_i=\sum_jA_{ij}v_j$$

Mathematical statement

For a compact Lie group $G$, let $\mu_G$ denote normalized Haar measure, so $\int_Gd\mu_G(g)=1$. Given a base similarity $s(q,k)$, define the group-integrated invariant affinity $\bar{s}(q,k)=\int_Gs(q,g\cdot k)\,d\mu_G(g)$. A practical Monte Carlo approximation uses $g_1,\ldots,g_K\sim\mu_G$: $\bar{s}_K(q,k)=K^{-1}\sum_{r=1}^Ks(q,g_r\cdot k)$. For Gaussian attention logits, use $s(q,k)=-\|q-k\|^2/(2\tau^2)$, where $\tau$ is a temperature, or exponentiate after averaging to obtain a positive kernel $K_G(q,k)=K^{-1}\sum_r\exp[-\|q-g_r\cdot k\|^2/(2\tau^2)]$. The paper identifies integration over $G$ as one of three invariant-kernel classes and proves that graph operators built from such kernels converge to operators on the quotient $\mathcal{M}/G$. Thus the attention logits discard orbit-direction distinctions while retaining quotient geometry.

Implementation notes

1. Integration point: modify the attention-logit computation in one or more heads. Assume queries and keys contain a subvector on which the known group action is defined; for images this can be a feature map transformed by differentiable spatial rotation, while for point clouds it can be a rotated coordinate or steerable feature block. 2. Pseudocode: sample or deterministically select $K$ group elements; for each query-key pair compute $z_{ijr}=-(\|q_i-g_r\cdot k_j\|^2)/(2\tau^2)$; either use $\ell_{ij}=\log\operatorname{meanexp}_r(z_{ijr})$ for numerically stable positive-kernel attention, or $\ell_{ij}=K^{-1}\sum_rz_{ijr}$; apply row-wise softmax to $\ell_{ij}$, then multiply by values. Cache transformed keys $g_r\cdot k_j$ within a layer so the extra group dimension is computed once. 3. Haar integration and invariance are mathematical; Monte Carlo variance, $K$, temperature $\tau$, and whether values are transformed are engineering choices. Start with a fixed rotation grid, then test random Haar samples and antithetic pairs $(g,g^{-1})$. Use standard attention for heads not assigned symmetry-aware processing. 4. First cheap experiment: a 4-layer small ViT on rotated MNIST or CIFAR-10, with $K=4,8,16$ SO(2) samples and the same parameter count as standard attention. Evaluate clean and unseen-angle accuracy, attention consistency between an image and its rotated copy, and training speed. A useful signal is substantially lower rotation-induced prediction variance and improved unseen-angle accuracy at equal augmentation budget; reject the method if the extra $K$-fold attention cost does not yield a better accuracy-per-FLOP tradeoff.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified Haar-averaged SO(2) Gaussian attention in verify_haar_attention.py. All three mechanism predictions manifested: dense Haar orbit variation was 2.3e-17, Monte Carlo error had observed slope -0.511 versus predicted -0.5, and orbit variation stayed at numerical precision across temperatures. In the toy attention test with keys rotated and queries fixed, output RMSE fell from 0.0869 for standard attention to 1.47e-13 for K=32 Haar attention; this confirms the claimed invariance mechanism, but not an accuracy or efficiency win.

Agent confidence: 9/10

Baseline: Standard Gaussian attention: key-rotation output RMSE 0.0868914; orbit affinity standard deviations for tau=[0.3,0.5,0.8,1.2] were [0.25635,0.31517,0.32335,0.25037].
Idea: K=32 deterministic SO(2) Haar-grid attention: key-rotation output RMSE 1.47e-13, approximately 5.9e11x lower; dense-grid orbit std 2.30e-17; Monte Carlo RMS error decreased from 0.2245 at K=1 to 0.01919 at K=128.

Limitations: Only a NumPy toy kernel/attention experiment was tested; no neural-network training, image benchmark, learned values, random-Haar versus deterministic-grid comparison, GPU implementation, wall-clock/FLOP accounting, or accuracy-per-compute evaluation was performed.

How to run: python3 verify_haar_attention.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.08987) Sep 2, 2026

Ahead of the baseline (−1.7%) but not statistically significant (p=0.329, wins 6 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.2809
Idea mean: 0.2762
Effect (Δ): -0.0047 (−1.7%; negative = idea better)
Wins: 6 / 8 paired seeds
p-value: 0.329 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in sequence forecasting track was tested; no vision/image rotation track, random Haar sampling, antithetic sampling, transformed values, wall-clock/FLOP accounting, or larger models were evaluated. The local implementation averages rotations over learned feature-coordinate pairs, which is a practical SO(2) proxy rather than a domain-calibrated image-space group action.

How to run: python3 bench_haar_attention.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented Haar-averaged SO(2) Gaussian attention in a local transformer for the structurally matched sequence forecasting track, using the benchmark's train_model, sweep, paired seeds, and permutation test. The idea achieved mean test MSE 0.27622 versus baseline 0.28088, delta -0.00466, but p=0.32895, so the required significant-win criterion was not met. Trained-model key-rotation logit sensitivity decreased from 2.9819 to 1.5140, a real but incomplete invariance signal; the quantitative mechanism signature was therefore not confirmed under the predeclared threshold.

Baseline: Sequence transformer, best shared-grid config lr=0.004, tau=1.0, K=16; 8-seed test MSE mean 0.280876, std 0.037198.
Idea: Haar-averaged sequence transformer, best config lr=0.004, tau=1.0, K=16; 8-seed test MSE mean 0.276219, std 0.038185; paired delta -0.004658, 6/8 wins, permutation p=0.32895.
Mechanism signature: Mean absolute trained attention-logit change after a 1.137-radian key rotation: baseline 2.981916, idea 1.514043; reduction factor 1.97x, below the 4x confirmation threshold; confirmed=false.

Benchmark result