Replace consecutive or randomly assigned transformed KV coefficients with groups whose variance-volume is approximately equal. Train one equal-size vector-quantizer codebook per group, so a fixed-width cache does not waste its low-rate budget by forcing high-variance and low-variance coordinates into badly mismatched groups. This is a drop-in quantization-layout change that can be applied to keys, values, or both.
Replace dense cross-attention weights with a balanced transport plan whose nonzero query-key edges are maintained by a multiscale active-set procedure. Solve the coarse token-group problem first, lift its support to the fine token grid, add only edges indicated by local cost or marginal residuals, and warm-start the fine problem from the lifted plan. This should provide a principled sparse attention pattern rather than fixing a global top-k pattern before seeing the transport solution.
Compress a transformer KV cache by selecting actual past tokens whose key or hidden-state columns form a stable basis for all cached tokens. Instead of retaining tokens with the largest attention scores or leverage scores independently, compute rank-revealing pivoting of the leading right-singular-vector matrix and retain its pivot columns, then evaluate attention using the representatives plus an optional low-cost residual correction.
Replace fixed-budget token or patch pruning with greedy selection that combines a teacher-derived relevance score and Gaussian-process mutual information. Select an item when it is both relevant and non-redundant, and stop when the largest remaining information gain falls below a calibrated threshold instead of retaining a fixed number of items.
Use the number of tokens participating in an attention operation to adapt the projected query/key dimension according to the sharp Johnson–Lindenstrauss bound. Short attention windows receive a dimension limited by n-1, while longer windows use a logarithmic dimension, avoiding a fixed large head dimension when the current attention set contains too few points to require it.
Replace uniform or random landmark selection in low-rank attention with approximate greedy pivoted QR selection on the attention matrix. At each step, select the key whose residual attention column has the largest norm, orthogonalize it against previously selected columns, and stop when the residual energy falls below a target tolerance. Use the selected columns as an adaptive attention basis and reconstruct the full attention output through a small coefficient matrix.
Turn attention weights into a Boolean support scenario and prune edges using local-surjectivity constraints rather than independently thresholding each row. Preserve at least one compatible continuation for every local window, then favor a strongly connected support graph so pruning removes redundant mixtures while maintaining a globally coherent sparse attention pattern.
Construct a sparse attention support by solving multiple small perturbed assignment problems between query and key embeddings and taking the union of the selected optimal matchings. Use the resulting spanning tree as the only set of cross-token edges, with edge biases determined by empirical assignment frequency.
Use the dimension-specific relation A_3=0 to remove all intermediate channels transforming as the third exterior power of the two-dimensional vector representation. In tensor-product attention or equivariant MLPs, this is an exact algebraic pruning rule rather than approximate low-rank compression.
Convert attention scores into binary incidence set systems at one or several score thresholds, then group queries that induce the same sampled-key pattern. Compute the expensive key-value aggregation once per pattern and reuse it for all queries in the group. The method is exact for the thresholded routing component and approximates dense attention when queries have a small number of stable high-weight neighborhoods.