Residual-pivoted landmark attention

Integrate this at the attention-kernel stage for a noncausal self-attention layer, initially on one head. Let $A=\operatorname{softmax}(QK^\top/\sqrt d)\in\mathbb{R}^{N\times N}$ and $V\in\mathbb{R}^{N\times d_v}$. Build a landmark set $J$ of at most $r$ keys. Initialize an implicit residual $R=A$ and an empty basis $Q_r$. At each step, estimate $\|R_{:,j}\|_2$ for candidate keys, choose the largest candidate, append $q=R_{:,j}/\|R_{:,j}\|_2$ to the basis, and update candidate residuals by $R_{:,j}\leftarrow R_{:,j}-q(q^\top R_{:,j})$. Stop when the largest estimated residual norm is below $\tau\|A\|_F$, or when $r=r_{\max}$. Form $C=A_{:,J}$ and compute a ridge-stabilized coefficient matrix $T=(C^\top C+\lambda I)^{-1}C^\top A$. Replace exact attention output $AV$ with $C(TV)$. The residual recursion and greedy criterion come from the paper; $\eta$, tolerance $\tau$, candidate subsampling rate, and ridge parameter $\lambda$ are empirical engineering choices. To avoid materializing all of $A$, compute selected columns and candidate residual probes on demand, or apply the procedure to a feature-based approximation of the attention kernel. First test a 2-layer Transformer encoder on WikiText-2 or a synthetic attention matrix with controlled singular-value decay, comparing adaptive pivots with uniform and random Nyström landmarks at sequence lengths 512 and 2048. Record attention-output relative error $\|AV-C(TV)\|_F/\|AV\|_F$, validation perplexity, latency, and peak memory. Success is lower error at fixed rank or at least 2x lower attention memory/FLOPs at matched perplexity.