# Co-Prime Virtual-Aperture Attention

- ID: 3030
- Canonical URL: https://synthcore.org/idea/3030/co-prime-virtual-aperture-attention
- API JSON: https://synthcore.org/api/idea/3030.json
- API Markdown: https://synthcore.org/api/idea/3030.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2609.01979](https://arxiv.org/abs/2609.01979)
- Category: architecture
- Solves: speedup, memory, scalability
- ML areas: attention, transformer, inference-speedup, memory
- Math tags: combinatorics, linear-algebra, number-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace dense or single-dilation sparse attention with two sequential sparse attention stages whose offsets form co-prime arithmetic progressions. The first stage mixes tokens separated by multiples of M2, the second by multiples of M1; their composition reaches virtual offsets mM2+nM1, providing many structured long-range interactions from only M1+M2-1 physical offset families. Use causal masking and residual connections so the module can replace a standard transformer attention block without changing sequence length.

## Mathematical statement

The paper defines two physical sensing position sets with coprime positive integers M1 and M2. Their shared origin gives Ms=M1+M2-1 distinct physical positions, while the cross-sum virtual array is V_CPA={(mM2+nM1)d | m=0,...,M1-1, n=0,...,M2-1}, where d is the base spacing. We transplant the cross-sum structure to token offsets by setting d=1 and using Dt={mM2} and Dr={nM1}. For a sequence x0,...,xL-1, let SparseAttn_D(X)i attend only to valid indices j=i+delta for delta in D, with learned relative bias b_delta and standard scaled dot-product softmax weights. The proposed block is H=SparseAttn_Dr(SparseAttn_Dt(X)), followed by a residual output projection. Its effective receptive-field offsets are sums delta_t+delta_r=mM2+nM1, the virtual positions from the paper. Coprimality gcd(M1,M2)=1 is the structural condition intended to reduce repeated offsets; boundary clipping and accidental duplicate offsets are handled by deduplicating the offset set or retaining multiple computational paths.

## Key formulas

- $$M_s=\left|\mathcal{P}_{\rm t}\cup\mathcal{P}_{\rm r}\right|=M_1+M_2-1.$$
- $$\mathcal{V}_{\rm CPA}=\{(mM_2+nM_1)d\mid m=0,\ldots,M_1-1,\;n=0,\ldots,M_2-1\}.$$
- $$\operatorname{Attn}_{D}(X)_i=\sum_{\delta\in D_i}\operatorname{softmax}_{\delta}\!\left(\frac{q_i k_{i+\delta}^{\top}}{\sqrt{d_h}}+b_{\delta}\right)v_{i+\delta},\qquad D_t=\{mM_2\}_{m=0}^{M_1-1},\quad D_r=\{nM_1\}_{n=0}^{M_2-1}.$$
- $$H=\operatorname{Attn}_{D_r}\!\left(\operatorname{Attn}_{D_t}(X)\right),\qquad Y=X+W_oH,\quad \Delta_{\rm virtual}=mM_2+nM_1.$$

## Implementation notes

Integrate this as an alternative to one standard self-attention block in a decoder-only transformer. Choose small coprime integers, for example M1=3 and M2=4, and define signed offsets for bidirectional models or nonnegative offsets for causal models. For each layer, construct Dt={0,M2,2M2,...,(M1-1)M2} and Dr={0,M1,2M1,...,(M2-1)M1}; for every batch, head, and token i, gather only positions i+delta that are inside the sequence and respect the causal mask. Apply ordinary scaled dot-product attention independently in stage t and stage r, using separate Q/K/V projections or shared projections, then add a residual output projection and the normal MLP. Compact pseudocode is: `Dt=offsets(M2,M1); Dr=offsets(M1,M2); Z=sparse_attn(X,Dt); H=sparse_attn(Z,Dr); Y=X+Wo(H)`. The paper supplies the physical-count formula and cross-sum construction; no mathematical quantities need to be estimated during training. Enumerate all offset pairs (dt,dr) to measure the effective virtual-offset set and deduplicate sums, while separately counting multiply-adds, memory reads, and kernel launches. Start with a 6-layer, 256-dimensional character-level or WikiText-2 transformer at sequence length 1024. Compare dense attention, local-window attention with the same edge count, standard single-dilation attention, and the co-prime block. Report validation perplexity at equal training FLOPs, wall-clock throughput, peak activation memory, and the fraction of tokens receiving information from offsets above 128. The success signal is equal-or-lower perplexity with at least 2x lower attention cost, or substantially larger long-range coverage at the same edge budget. Ablate coprime pairs against non-coprime pairs such as (2,4), testing whether virtual-offset count and quality degrade as predicted.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built a causal co-prime virtual-aperture attention MVP with M1=3 and M2=4, plus sparse and dense baselines. The math check reproduced 6 physical offset families, 12 unique virtual offsets, exact graph reachability, and fewer virtual offsets for the non-coprime (2,4) control. On a seeded lag-17 copy task, CPA reached 99.83% accuracy and loss 0.0201 at the same 7 attention edges per token as the sparse baseline, which reached 4.80% accuracy and loss 2.9467; this is a clear toy-task signal for long-range coverage, but not a speed win.

### Mechanism check

- Verdict: Built a causal co-prime virtual-aperture attention MVP with M1=3 and M2=4, plus sparse and dense baselines. The math check reproduced 6 physical offset families, 12 unique virtual offsets, exact graph reachability, and fewer virtual offsets for the non-coprime (2,4) control. On a seeded lag-17 copy task, CPA reached 99.83% accuracy and loss 0.0201 at the same 7 attention edges per token as the sparse baseline, which reached 4.80% accuracy and loss 2.9467; this is a clear toy-task signal for long-range coverage, but not a speed win.
- Confidence: 8/10
- Limitations: This is a tiny synthetic one-layer experiment with sequence length 24, one coprime pair, one non-coprime math control, and 180 optimization steps. It does not test language-model perplexity, multi-layer transformers, realistic data, equal FLOPs including projection costs, optimized sparse kernels, activation memory, or generalization beyond the fixed lag-17 task; measured wall-clock time was not improved.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.3787
- Paired wins: 5/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/1219/bench_report.json)
- [coprime_attention_experiment.py](https://synthcore.org/code/1219/coprime_attention_experiment.py)
- [report.md](https://synthcore.org/code/1219/report.md)
- [report_bench_2026-09-03T123435.md](https://synthcore.org/code/1219/report_bench_2026-09-03T123435.md)
- [results.json](https://synthcore.org/code/1219/results.json)
- [stage2_bench.py](https://synthcore.org/code/1219/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/1219)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.