Replace token-by-token KV storage after an SSM or recurrent encoder with an online allocate-on-novelty cache. A new slot is created only when the incoming key is sufficiently dissimilar from every stored key; otherwise the incoming value is merged into its nearest slot, so repeated or redundant content does not grow the cache.
Replace ordinary per-channel or per-token KV quantization with a structured orthogonal transform followed by blockwise 3-bit quantization. Use a normalized Walsh-Hadamard transform and small SO(4) rotations to spread outliers across coordinates, quantize the transformed vectors, and exploit orthogonality to rotate queries and attention outputs so unquantized attention remains mathematically equivalent.
Construct a sparse attention support by solving multiple small perturbed assignment problems between query and key embeddings and taking the union of the selected optimal matchings. Use the resulting spanning tree as the only set of cross-token edges, with edge biases determined by empirical assignment frequency.