A JoLT for the KV cache: Near-lossless KV cache compression via joint Lagrangian allocation of Tucker ranks and a rotated residual for llms
arXiv:2607.12550
2026
Memory
2 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper treats the KV cache as a tensor rather than as independent matrices, preserving the head and layer axes while factorizing token and feature axes with a partial Tucker representation. Its transferable asset is the explicit rate-distortion tradeoff between Tucker rank and a rotated, low-bit residual: ranks remove structured energy, while rotation makes the remaining error more uniformly quantizable. A practical deployment is to solve one byte-constrained allocation problem per layer group and separately for keys and values, instead of applying a single rank or bit-width globally. This is promising for long-context inference because it can reduce KV memory while preserving cache fidelity.
Ideas from this paper
△ Mechanism confirmed, baseline not beaten
2026
Compress each layer's KV tensor with a partial Tucker approximation over token and feature axes, then encode the truncation residual with a rotated uniform quantizer. Select token rank, feature rank, and residual bit-width jointly under a global byte budget, allowing values with flat spectra to receive residual bits while keys may receive more low-rank capacity.
Useful8/10
Difficulty6/10
Novelty6/10
✗ Failed on benchmark
2026
Use a low-rank Tucker reconstruction as a structured backbone and quantize only its residual after an orthogonal rotation. The rotation preserves residual energy but redistributes it across coordinates, reducing dynamic-range imbalance and making 2- or 4-bit uniform quantization less damaging than direct quantization of the original KV tensor.
Useful7/10
Difficulty5/10
Novelty6/10