RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

What the math gives to ML

The paper offers a transferable memory-compression design for attention: apply a structured orthogonal mixing transform to KV vectors before low-bit scalar quantization, then perform attention without reconstructing full-precision cache tensors. The mathematical asset is preservation of dot products under orthogonal transforms, combined with Walsh-Hadamard mixing and small block-diagonal SO(4) rotations that can distribute heavy-tailed coordinates more evenly. This can reduce the worst-channel error that often dominates 3-bit KV quantization while requiring only fast structured transforms rather than dense learned rotations. The first implementation should target decoder-only attention with calibration-learned block rotations and a fused dequantize-and-matmul kernel, measuring perplexity, cache size, and decode bandwidth.