Spend Bits Where Queries Look: KV Cache Vector Quantization with Attention-Preserving Transforms

What the math gives to ML

The paper recasts KV-cache compression as attention-product distortion rather than ordinary entrywise reconstruction error. Its most directly transferable construction is a volume-aware partition of transformed coefficients: equal-volume groups allow equal-size vector codebooks to attain the variable-rate high-resolution optimum under a fixed-width cache layout. This is especially relevant for two-bit KV caches, where scalar quantization and variance-equalizing rotations waste bits on low-energy coordinates. The safest first implementation is to retain an existing transform such as PCA or a Hadamard rotation, replace consecutive coefficient groups with calibration-based equal-volume groups, and evaluate attention distortion and long-context retrieval at identical bits and bandwidth.