# Increasing-Motif Attention Mask

- ID: 2452
- Canonical URL: https://synthcore.org/idea/2452/increasing-motif-attention-mask
- API JSON: https://synthcore.org/api/idea/2452.json
- API Markdown: https://synthcore.org/api/idea/2452.md
- Verification status: unverified
- Source: [arXiv:2608.17354](https://arxiv.org/abs/2608.17354)
- Category: architecture
- Solves: scalability, generalization, memory
- ML areas: attention, transformer, memory, inference-speedup
- Math tags: combinatorics, graph-theory, algebra
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Parameterize a sequence attention graph by a finite graph motif on positions [r], then lift that motif to arbitrary sequence lengths using strictly increasing embeddings. Instead of learning an independent positional mask for every context length, the model learns a small collection of order-preserving edge types whose induced graph is valid at every length and can be extended without retraining.

## Mathematical statement

The paper studies an \(\Inc\)-invariant chain \(\mathcal G=(G_n)_{n\ge1}\), where \(G_n\) is a graph on ordered positions and strictly increasing maps preserve edges. The finite-stage principle is that the chain is determined by a finite index \(r=\operatorname{ind}(\mathcal G)\) and the base graph \(G_r\), up to increasing relabelings. The extracted lemma gives the concrete closure rule: if \(n\ge r\), \(u<v\), \(v\ge u+r\), and an edge \((k,l)\in E(G_n)\) satisfies \(k\le u<v\le l\), then \((u,v)\in E(G_n)\). Its proof uses the base-edge displacement condition \(0\le k-i\le l-j\le n-r\) for some \((i,j)\in E(G_r)\). In the adaptation, \(r\) is the learned motif width, \(E_r\) is a finite set of relative position pairs, and \(E_n\) is the generated attention support; the interval closure rule is optionally enabled as a deterministic augmentation.

## Key formulas

- $$0\le k-i\le l-j\le n-r,$$
- $$v-u\ge r\ \land\ (k,l)\in E(G_n)\ \land\ k\le u<v\le l\ \Longrightarrow\ (u,v)\in E(G_n),$$
- $$E_n^{\mathrm{motif}}=\{(\phi(i),\phi(j)):\ (i,j)\in E_r,\ \phi:[r]\hookrightarrow[n]\ \text{strictly increasing}\},$$
- $$A_{uv}=\begin{cases}(q_u^{\mathsf T}k_v)/\sqrt d+b_{\tau(u,v)},&(u,v)\in E_n,\\-\infty,&(u,v)\notin E_n,\end{cases}\qquad \operatorname{Attn}(u)=\sum_v\operatorname{softmax}_v(A_{uv})v_v.$$

## Implementation notes

Integrate this into the attention-logit mask of a small decoder-only Transformer. Choose a motif width r, for example r=8 or 16, and learn a binary or soft gate for each ordered pair (i,j) with 1<=i<j<=r, plus optional local self-loop and reverse-edge types. For a sequence of length n, generate candidate edges by matching the relative order pattern of each motif pair; do not enumerate all increasing maps. The simplest efficient parameterization uses edge offsets and endpoint distances: each base pair (i,j) creates edges (u,v) whose quantized normalized coordinates match that pair. Add the paper's closure rule as a second mask pass: whenever a selected edge (k,l) encloses (u,v) with v-u>=r, mark (u,v) selected. Implement interval closure using per-row difference arrays or a CUDA range-fill kernel, not pairwise Python loops. Pseudocode: initialize E_n from motif pairs; for each source k and destination l in E_n, range-add all destinations v in [k+1,l] satisfying v-k>=r; prefix-sum the ranges; form logits with negative infinity outside E_n. Train motif gates jointly with the model using a sparsity penalty on the number of edges. The mathematics fixes the lifting and closure rule; motif edge probabilities, thresholds, and approximate coordinate hashing are empirical choices. First test on WikiText-103 or C4 with a 6-layer, 256-hidden Transformer at context lengths 512, 1024, and 2048 against dense attention and fixed sliding-window attention at matched FLOPs. Success means lower perplexity than local attention at equal attention-edge count, stable extrapolation at lengths unseen during training, and attention memory scaling with the generated edge count rather than n squared.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.