△ Mechanism confirmed, baseline not beaten 2026

Entropy-constrained unbiased block quantization

Implementation & benchmark of arXiv:2608.18147 — Entropy-Constrained Adaptive Stochastic Quantization

Usefulness8/10

Difficulty6/10

Novelty6/10

Memory Memory Speedup Accuracy Quantization Transformer Kv cache Inference speedup Information theory Optimization Probability Dynamic programming

Source paper: Entropy-Constrained Adaptive Stochastic Quantization arXiv:2608.18147 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace fixed-grid stochastic quantization of each tensor block with an adaptive ordered codebook selected under both an entropy budget and a maximum number of codepoints. Within every interval between adjacent codepoints, use unbiased stochastic interpolation, so the quantized block remains unbiased while the emitted symbol distribution becomes easier for arithmetic or Huffman coding to compress. The representation should reduce actual compressed bytes at fixed MSE, or reduce MSE at fixed compressed size.

Formulas

$$\min_{Q\subseteq P} MSE(Q,X)\quad\text{subject to}\quad |Q|\le s,\;H(\widehat X)\le b,\;\mathbb{E}[\widehat X]=X.$$

$$\Pr(\widehat x=q_i\mid x)=\frac{q_{i+1}-x}{q_{i+1}-q_i},\qquad \Pr(\widehat x=q_{i+1}\mid x)=\frac{x-q_i}{q_{i+1}-q_i},\quad x\in[q_i,q_{i+1}].$$

$$\mathbb{E}[\widehat x\mid x]=x,\qquad \mathbb{E}[(\widehat x-x)^2\mid x]=(q_{i+1}-x)(x-q_i).$$

$$\mathcal{L}(Q;\lambda)=\sum_{t=1}^{d}(\widehat x_t-x_t)^2+\lambda d\left[-\sum_{j=1}^{k}\pi_j\log_2\pi_j\right],\qquad H(\widehat X)=-\sum_{j=1}^{k}\pi_j\log_2\pi_j.$$

Mathematical statement

The paper defines ECASQ for a real vector X, permissible alphabet P, maximum codebook size s, and average entropy budget b as minimizing MSE(Q,X) subject to |Q|<=s, H(hat X)<=b, and E[hat X]=X. Here Q is the selected scalar codebook, hat X is the random quantized vector, and H is Shannon entropy in bits per entry. For ordered codepoints q_1<...<q_k, an input x in [q_i,q_{i+1}] is mapped to q_i with probability (q_{i+1}-x)/(q_{i+1}-q_i) and to q_{i+1} with probability (x-q_i)/(q_{i+1}-q_i). This gives E[hat x|x]=x and conditional MSE E[(hat x-x)^2|x]=(q_{i+1}-x)(x-q_i). If pi_j is the empirical probability that output symbol q_j is emitted, the rate is H(hat X)=-sum_j pi_j log_2(pi_j). The constrained problem can be searched through the Lagrangian D(Q)+lambda*d*H(Q), where D is total MSE, d is block length, and lambda>=0 is a rate multiplier. The paper reports an approximate dynamic program using O(s d^2) time and O(d) space, with MSE no larger than the optimum using one fewer entropy bit per entry, followed by iterative refinement.

Implementation notes

Integrate this at the tensor-block quantization boundary, initially for 1D blocks of gradients, weights, activations, or KV-cache rows. Use blocks of d=64 or 128 values and retain the original block's scale/range normalization if needed. For each block: (1) sort or histogram values and construct a candidate alphabet P from uniformly spaced points, percentile points, or observed values plus endpoints; (2) for a candidate ordered codebook Q={q_1,...,q_k}, assign each x_t to its bracketing interval and compute stochastic probabilities from the interpolation formula; (3) estimate output probabilities by expected emissions, pi_j=(1/d) sum_t Pr(hat x_t=q_j|x_t), rather than from one random sample; (4) compute distortion D=sum_t(q_{i(t)+1}-x_t)(x_t-q_{i(t)}) and rate R=-sum_j pi_j log2 pi_j; (5) select Q minimizing D+lambda*d*R among k<=s. An MVP can approximate the paper's dynamic program with states dp[k,j], the minimum objective for using k codepoints through candidate index j, and transitions over previous cutpoints; interval distortion and symbol probabilities should be computed with prefix sums. Binary-search lambda until R is near target b, then run 2-5 local refinement passes that move codepoints to nearby candidates and accept reductions in D+lambda*d*R. During compression, sample symbols once and entropy-code the integer stream with arithmetic coding; transmit codebook and block metadata. The mathematical quantities are unbiased interpolation, conditional MSE, entropy, and the Lagrangian; candidate alphabets and lambda search are engineering choices. First test on GPT-2-small or a 6-layer decoder, quantizing weights or activations at 3-8 symbols and comparing uniform stochastic quantization plus arithmetic coding. Measure perplexity, realized bits per value, MSE, and encode/decode throughput. Success is at least 15% fewer post-coded bytes at equal perplexity, or lower perplexity at equal bytes, without material quantization-latency growth.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built an entropy-constrained unbiased block quantizer with ordered stochastic interpolation, expected-emission entropy, exhaustive small-codebook Lagrangian search, and sampling. The core MSE identity matched to 6.1e-17; 5,000-sample unbiasedness checks had maximum empirical bias 0.0122, consistent with finite-sample error, and selected entropy was monotone as lambda increased. On the skewed toy block, adaptive codebooks reduced MSE by 20.7% at a nearby 1.88-bit/value rate and by 54.8% at a low-rate comparison, but actual arithmetic-coded bytes and neural-model quality were not tested.

Agent confidence: 8/10

Baseline: Uniform grids: rate/MSE = 0.972/37.282 (2 points), 1.501/11.543 (3), 1.878/4.518 (4), 2.088/2.351 (5).
Idea: Adaptive selected sweep: lambda 0 to 1.2 produced rates 2.147, 2.147, 2.147, 2.064, 1.802, 1.384, 1.226 bits/value with MSE 2.201, 2.201, 2.201, 2.373, 3.581, 11.667, 16.841; rate was monotone. Exact conditional MSE error was 6.1e-17; maximum empirical bias was 0.0122.

Limitations: The candidate search is exhaustive and intended only for small toy blocks; no dynamic-programming optimization, local refinement, arithmetic/Huffman coder, metadata overhead, throughput measurement, tensor integration, or neural-network perplexity/accuracy experiment was performed.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.18147) Sep 2, 2026

No significant difference vs. the tuned baseline (1.5451 metric points, p=0.5494, wins 5 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0
Idea mean: 6.5854
Effect (Δ): 1.5451 (n/a; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.5494 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: No effect
Baseline tuning: swept over 0 configs

Limitations:

The run used a compact six-epoch training budget and a three-configuration parity grid rather than the full default epoch budget. It evaluated task MSE but did not implement arithmetic/Huffman coding, metadata overhead, compressed byte measurements, throughput, activation quantization, or larger language-model perplexity.

How to run: python3 bench_exp.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented adaptive stochastic block quantization for a transformer_tiny on the structurally matched sequence track, with a uniform stochastic quantization baseline and shared learning-rate/level sweep. The mechanism signature was confirmed on trained model weights, with observed mean absolute quantization bias 0.003634. However, adaptive quantization worsened test MSE: 6.58545 versus 5.04036, paired delta +1.54509, permutation p=0.5494; therefore there was no significant win.

Baseline: sequence/transformer_tiny; best lr=0.0015, levels=4; 8-seed test MSE mean=5.040362, std=3.954218
Idea: adaptive stochastic block quantization; best lr=0.0015, levels=4; 8-seed test MSE mean=6.585448, std=5.840950
Mechanism signature: trained-model observed mean absolute single-draw quantization bias=0.003634; confirmed=true
Paired comparison: delta_mean=+1.545085; idea_wins=5/8; permutation_p=0.5494; verdict=no significant win

Benchmark result