Increasing-Motif Attention Mask

Integrate this into the attention-logit mask of a small decoder-only Transformer. Choose a motif width r, for example r=8 or 16, and learn a binary or soft gate for each ordered pair (i,j) with 1<=i<j<=r, plus optional local self-loop and reverse-edge types. For a sequence of length n, generate candidate edges by matching the relative order pattern of each motif pair; do not enumerate all increasing maps. The simplest efficient parameterization uses edge offsets and endpoint distances: each base pair (i,j) creates edges (u,v) whose quantized normalized coordinates match that pair. Add the paper's closure rule as a second mask pass: whenever a selected edge (k,l) encloses (u,v) with v-u>=r, mark (u,v) selected. Implement interval closure using per-row difference arrays or a CUDA range-fill kernel, not pairwise Python loops. Pseudocode: initialize E_n from motif pairs; for each source k and destination l in E_n, range-add all destinations v in [k+1,l] satisfying v-k>=r; prefix-sum the ranges; form logits with negative infinity outside E_n. Train motif gates jointly with the model using a sparsity penalty on the number of edges. The mathematics fixes the lifting and closure rule; motif edge probabilities, thresholds, and approximate coordinate hashing are empirical choices. First test on WikiText-103 or C4 with a 6-layer, 256-hidden Transformer at context lengths 512, 1024, and 2048 against dense attention and fixed sliding-window attention at matched FLOPs. Success means lower perplexity than local attention at equal attention-edge count, stable extrapolation at lengths unseen during training, and attention memory scaling with the generated edge count rather than n squared.