△ Mechanism confirmed, baseline not beaten 2026

Orthogonally mixed 3-bit KV cache

Implementation & benchmark of arXiv:2608.08081 — RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention

Usefulness7/10

Difficulty6/10

Novelty6/10

Memory Memory Inference speedup Accuracy Attention Kv cache Quantization Inference speedup Linear algebra Harmonic analysis Geometry Optimization

Source paper: RotaryQuant: Fitting 120B MoE Models on Consumer Hardware via Fused Compressed-Space Attention arXiv:2608.08081 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace ordinary per-channel or per-token KV quantization with a structured orthogonal transform followed by blockwise 3-bit quantization. Use a normalized Walsh-Hadamard transform and small SO(4) rotations to spread outliers across coordinates, quantize the transformed vectors, and exploit orthogonality to rotate queries and attention outputs so unquantized attention remains mathematically equivalent.

Formulas

$$M_{\mathrm{total}}=O_E\circ C_{KV}\circ Q_W$$

$$H_d^\top H_d=I_d,\qquad B=\operatorname{diag}(R_1,\ldots,R_{d/4}),\qquad R_i^\top R_i=I_4,\ \det(R_i)=1,\qquad U=BH_d$$

$$q^\top k=(Uq)^\top(Uk),\qquad \operatorname{Attn}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V$$

$$s=\frac{\max_j|x_j|+\epsilon}{2^{b-1}-1},\qquad \hat{x}=s\,\operatorname{clip}\!\left(\operatorname{round}(x/s),-2^{b-1}+1,2^{b-1}-1\right),\quad b=3$$

Mathematical statement

The paper composes compression as $M_{\mathrm{total}}=O_E\circ C_{KV}\circ Q_W$, where $Q_W$ is weight quantization, $C_{KV}$ is KV-cache compression, and $O_E$ is expert offloading; this identifies $C_{KV}$ as an independent module applicable to ordinary dense attention. Let $d$ be the head dimension, assumed divisible by $4$. Let $H_d$ be the normalized Walsh-Hadamard matrix satisfying $H_d^\top H_d=I_d$. Let $R_i\in SO(4)$ be four-dimensional rotations satisfying $R_i^\top R_i=I_4$ and $\det(R_i)=1$, and define $B=\operatorname{diag}(R_1,\ldots,R_{d/4})$ and $U=BH_d$. Then $U^\top U=I_d$. For query and key vectors $q,k\in\mathbb{R}^d$, orthogonality gives $(Uq)^\top(Uk)=q^\top k$, so both can be transformed without changing unquantized attention logits. For values $v$, transformed values $v'=Uv$ can be converted back after attention using $U^\top$. The scalar quantizer for a block $x\in\mathbb{R}^m$ uses scale $s=(\max_j|x_j|+\epsilon)/(2^{b-1}-1)$ and reconstruction $\hat{x}=s\operatorname{clip}(\operatorname{round}(x/s),-2^{b-1}+1,2^{b-1}-1)$, with $b=3$. The transform is valuable when it reduces block maxima or tail magnitudes, thereby reducing quantization error at fixed bit width.

Implementation notes

Integrate this at the decoder self-attention KV-cache write and read paths. For each attention head with dimension d divisible by four, implement a normalized fast Walsh-Hadamard transform using the recursive butterfly algorithm, then apply a block-diagonal collection of SO(4) matrices. Start with fixed random rotations, and calibrate them on a few thousand representative sequences by minimizing reconstruction MSE after 3-bit quantization; parameterize each SO(4) block with a normalized quaternion or use fixed rotations for the first MVP. On every new token, compute k1 = U(k) and v1 = U(v), partition them into quantization blocks, calculate one scale per block, round to signed 3-bit integers, and pack the codes into bytes. Store only packed codes and scales. At read time compute q1 = U(q), unpack and scale keys inside the attention dot-product kernel, calculate logits from q1 and reconstructed transformed keys, apply softmax, multiply by reconstructed transformed values, and rotate the output back with U.T. Pseudocode: U=block_SO4(FWHT); k1=U(k); v1=U(v); sk=maxabs(k1,blocks)/(22-1); sv=maxabs(v1,blocks)/(22-1); Kcache.append(pack3(round(k1/sk)),sk); Vcache.append(pack3(round(v1/sv)),sv); q1=U(q); logits=q1@dequant_fused(Kcache); y1=softmax(logits)@dequant_fused(Vcache); y=U.T@y1. Fuse unpacking, scale multiplication, and dot-product accumulation so full FP16 K/V tensors are never materialized. Orthogonality and the dot-product identity are exact mathematical properties; rotation selection, block size, and scales are estimated empirically. First test a 1B-3B decoder on WikiText-2 or C4 at 4K and 32K context against FP16 KV, standard per-token 3-bit KV, Hadamard-only 3-bit KV, and Hadamard-plus-SO(4) 3-bit KV. Success means at least 4x KV-memory reduction, perplexity degradation below 0.1, and reduced attention memory bandwidth; a stronger result is near-baseline perplexity with improved decode tokens per second.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a NumPy MVP implementing normalized FWHT, block-diagonal SO(4) mixing, blockwise signed 3-bit quantization, dequantized attention, and 3-bit packing. The math checks passed: SO(4) Gram error was 8.9e-16, transformed dot-product error 7.1e-15, inverse error 1.6e-15, and MSE divided by amplitude squared varied by only 5.4e-12, confirming orthogonal invariance and quadratic error scaling. Mixing reduced the worst block maximum from 6.0 to 2.67 with Hadamard and 3.05 with Hadamard+SO(4), while SO(4) attention relative RMSE improved from 0.2030 to 0.1772; however, raw outlier reconstruction MSE worsened and storage reduction was only 1.45x versus FP16 with fp32 scales.

Agent confidence: 8/10

Baseline: Identity 3-bit: outlier MSE 0.01312; attention relative RMSE 0.20296; worst block max 6.00; approximately 1.45x storage reduction versus FP16.
Idea: Hadamard+SO(4) 3-bit: outlier MSE 0.01790; attention relative RMSE 0.17721; worst block max 3.05; transformed dot-product error 7.1e-15; amplitude-normalized MSE relative spread 5.4e-12. Hadamard-only achieved worst block max 2.67 and attention relative RMSE 0.20276.

Limitations: Only toy NumPy experiments were performed. No real decoder, WikiText-2 or C4 perplexity, long-context cache benchmark, CUDA fused kernel, rotation calibration, throughput measurement, or lower-precision scale accounting was tested.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.08081) Sep 2, 2026

Ahead of the baseline (−0.4%) but not statistically significant (p=0.8602, wins 3 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.2784
Idea mean: 0.2772
Effect (Δ): -0.0012 (−0.4%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.8602 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±4.2%
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

This is a small 400/200-sample sequence task with a 12-epoch tiny transformer, not a pretrained decoder or long-context inference benchmark. It does not measure packed-cache memory bandwidth, fused CUDA kernels, throughput, rotation calibration, or FP16 scale storage.

How to run: python3 bench_kv.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented explicit sequence-level self-attention with normalized FWHT, fixed SO(4) rotations, blockwise signed 3-bit K/V quantization, and inverse rotation. The idea obtained mean test MSE 0.277234 versus baseline 0.278424, but paired delta was -0.001190 with p=0.8602 and only 3/8 wins; worked=false. The trained-model mechanism signature observed 0.12767 relative K/V reconstruction error and 0.04758 relative logit perturbation, confirming quantization perturbation but not a significant task improvement.

Baseline: Best lr=0.003; mean test MSE 0.278424, std 0.033547; sweep means lr=0.001: 0.325838, lr=0.003: 0.285828, lr=0.006: 0.311283.
Idea: Best lr=0.003; mean test MSE 0.277234, std 0.042793; paired delta=-0.001190, p=0.8602, 3/8 wins; idea sweep means lr=0.003: 0.277234, lr=0.001: 0.315460, lr=0.006: 0.311577.

Benchmark result