Entropy-Constrained Adaptive Stochastic Quantization

What the math gives to ML

The paper turns unbiased stochastic quantization into a rate-distortion optimization problem: the quantizer must minimize MSE while simultaneously controlling the entropy of emitted symbols and the codebook size. The transferable asset is that unbiasedness can be preserved exactly by stochastic interpolation between adjacent adaptive codepoints, while a dynamic-programming or Lagrangian search chooses codepoints that are favorable to subsequent entropy coding. This is especially promising for tensor blocks, gradients, and KV caches, where symbol-frequency skew can make true compressed size much smaller than nominal bitwidth. A practical first transfer is an entropy-aware block quantizer with a dual-updated rate multiplier, evaluated at equal post-entropy-coding bytes and equal downstream model quality.