Balanced design attention

Integrate the method at the self-attention operation of a Transformer encoder or decoder. Let the sequence length be $v$ and choose a block design with $b$ blocks of size $k$; store each block as an integer index list and store inverse incidence lists $\mathcal B(i)=\{j:i\in B_j\}$, each of size exactly $r$. At each sparse attention layer: (1) compute the usual projected tensors $Q,K,V$; (2) for every block $B_j$, gather $Q_{B_j},K_{B_j},V_{B_j}$; (3) compute ordinary scaled dot-product attention only among the $k$ tokens in that block; (4) scatter the output back to the original token positions; and (5) average the $r$ outputs received by each token, or concatenate them and use a learned output projection. Compact pseudocode is Y=zeros(v,d); C=zeros(v); for j in 1..b: idx=B[j]; S=softmax(Q[idx] @ K[idx].T / sqrt(d_h)); Y[idx]+=S @ V[idx]; C[idx]+=1; Y/=C. The incidence formula determines the mask: M[i,j]=1 iff i in B[j]. Verify offline that every row sum is $r$ and every pairwise co-occurrence sum is $\lambda$. The exact design statistics are computed from $M$; attention probabilities, loss, and throughput are measured empirically. First test a small 4-layer, 8-head Transformer on WikiText-2 at sequence lengths 512 and 1024, plus a synthetic long-range copy task. Compare dense attention, contiguous windows, random overlapping blocks, and design blocks at matched values of $bk^2$. Report validation perplexity, copy accuracy, attention FLOPs, wall-clock throughput, pair-coverage variance, and peak memory. A useful success criterion is at least 2x lower attention FLOPs with similar perplexity, and better long-range copy accuracy than random or contiguous sparse masks. For autoregressive inference, separately benchmark K/V-cache memory and tokens per second because overlapping blocks can increase cache reads by $r$.