# Contraction-Gauge Quantization

- ID: 1118
- Canonical URL: https://synthcore.org/idea/1118/contraction-gauge-quantization
- API JSON: https://synthcore.org/api/idea/1118.json
- API Markdown: https://synthcore.org/api/idea/1118.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.18745](https://arxiv.org/abs/2607.18745)
- Category: optimization
- Solves: accuracy, memory, speedup
- ML areas: quantization, mlp, transformer, cnn
- Math tags: linear-algebra, optimization, probability, convex-analysis
- Ratings: usefulness 8/10; difficulty 5/10; novelty 6/10

## Idea description

Before quantizing a matrix product, reparameterize its factors as A'=AT and B'=T^{-1}B, preserving the exact full-precision product while changing the quantization difficulty of each factor. Choose a positive diagonal T=diag(t_1,...,t_K) that minimizes predicted post-quantization product error, rather than using output-channel scaling or a fixed heuristic grid. The gauge can be shared across several products when transformed-copy cost matters.

## Mathematical statement

The contraction-gauge identity is AB=(AT)(T^{-1}B), where A is in R^{m x K}, B is in R^{K x n}, and T is an invertible K x K matrix acting on the shared contraction index. Let A'=AT and B'=T^{-1}B. After quantization, let A_hat=A'+E_A and B_hat=B'+E_B, where E_A and E_B have mutually independent, zero-mean entrywise errors with variances vA_{ik}=E[E_A,ik^2] and vB_{kj}=E[E_B,kj^2]. Expanding A_hat B_hat-A'B' and using independence gives the exact expected squared Frobenius error E[||A_hat B_hat-A'B'||_F^2] = sum_{i,j,k}[vA_{ik}(B'_{kj})^2+(A'_{ik})^2 vB_{kj}+vA_{ik}vB_{kj}]. For a positive diagonal gauge T=diag(t_1,...,t_K), A'_{ik}=A_{ik}t_k and B'_{kj}=t_k^{-1}B_{kj}. Optimize J(t) over positive bounds l_k <= t_k <= u_k. The variances are analytic for subtractive dither or stochastic rounding and can be estimated from calibration data for deterministic round-to-nearest quantization.

## Key formulas

- $$AB=(AT)(T^{-1}B),\qquad T\in\mathrm{GL}(K).$$
- $$\widehat{A}=A'+E_A,\qquad \widehat{B}=B'+E_B,\qquad A'=AT,\qquad B'=T^{-1}B.$$
- $$\mathbb{E}\left[\left\|\widehat{A}\widehat{B}-A'B'\right\|_F^2\right]=\sum_{i=1}^{m}\sum_{j=1}^{n}\sum_{k=1}^{K}\left[v^A_{ik}(B'_{kj})^2+(A'_{ik})^2v^B_{kj}+v^A_{ik}v^B_{kj}\right].$$
- $$J(t)=\sum_{i,j,k}\left[v^A_{ik}(t)(t_k^{-1}B_{kj})^2+(A_{ik}t_k)^2v^B_{kj}(t)+v^A_{ik}(t)v^B_{kj}(t)\right],\qquad \ell_k\le t_k\le u_k.$$

## Implementation notes

(1) Integration point: modify the offline quantization pipeline for a linear layer or any explicit matrix product Y=AB. Do not alter the full-precision network. Insert a diagonal gauge on the contraction dimension, storing A'=A diag(t) and B'=diag(t)^{-1}B before calibration and quantization. At inference, multiply the two quantized factors as usual.

(2) Pseudocode:
```text
collect calibration tensors A and B
initialize z_k=0, with t_k=exp(z_k)
repeat for coordinate descent or bounded L-BFGS:
    Ap[i,k] = A[i,k] * exp(z[k])
    Bp[k,j] = exp(-z[k]) * B[k,j]
    quantize Ap and Bp or evaluate their quantizer residuals
    estimate vA[i,k] and vB[k,j]
    J = sum_i,j,k(vA[i,k]*Bp[k,j]^2 + Ap[i,k]^2*vB[k,j] + vA[i,k]*vB[k,j])
    update z while log(l_k) <= z_k <= log(u_k)
select the lowest-J gauge, then permanently quantize Ap and Bp
```
(3) Computed from the paper's mathematics: the transformed factors and the three-term product-error objective. Estimated empirically: quantizer variances, clipping scales, and useful gauge bounds. For RTN, use vA[i,k]=(Q(Ap[i,k])-Ap[i,k])^2 and the analogous expression for B; for stochastic rounding, average squared residuals over several random quantizations.

(4) First cheap experiment: quantize ResNet-18 linearized convolution products or a small transformer MLP on CIFAR-10 or WikiText-2 at 4 and 8 bits. Compare the identity gauge, a SmoothQuant-style baseline, and the optimized diagonal gauge at identical bitwidth and tensor count. Record predicted versus realized product MSE, end-to-end accuracy or perplexity, and calibration time. A successful result is at least 10 percent lower product MSE and improved 4-bit accuracy without increasing stored parameter count.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built a readable toy implementation of diagonal contraction-gauge quantization, including the exact AB=(AT)(T^{-1}B) check, the three-term expected-error objective, bounded coordinate optimization, and deterministic RTN evaluation. The mechanism manifested clearly: the derived optimum matched coordinate optimization to 2.5e-13 relative gap, Monte Carlo matched the analytic objective within 0.21%, and the predicted scaling laws t∝rho^0.5 and t∝vA^0.25 were observed exactly. On the fixed 4-bit toy product, the optimized gauge reduced realized product MSE by 18.86% versus the identity gauge.

### Mechanism check

- Verdict: Built a readable toy implementation of diagonal contraction-gauge quantization, including the exact AB=(AT)(T^{-1}B) check, the three-term expected-error objective, bounded coordinate optimization, and deterministic RTN evaluation. The mechanism manifested clearly: the derived optimum matched coordinate optimization to 2.5e-13 relative gap, Monte Carlo matched the analytic objective within 0.21%, and the predicted scaling laws t∝rho^0.5 and t∝vA^0.25 were observed exactly. On the fixed 4-bit toy product, the optimized gauge reduced realized product MSE by 18.86% versus the identity gauge.
- Confidence: 9/10
- Limitations: Only a small synthetic matrix product was tested, not a neural network or CIFAR/LM accuracy. The experiment used fixed global quantization steps and did not compare SmoothQuant, per-channel scales, calibration cost, storage overhead, or shared gauges across multiple products; deterministic RTN clipping effects were not systematically swept.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.3074
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_gauge.py](https://synthcore.org/code/215/bench_gauge.py)
- [bench_report.json](https://synthcore.org/code/215/bench_report.json)
- [gauge_quantization_experiment.py](https://synthcore.org/code/215/gauge_quantization_experiment.py)
- [report.md](https://synthcore.org/code/215/report.md)
- [report_bench_2026-08-30T205413.md](https://synthcore.org/code/215/report_bench_2026-08-30T205413.md)
- [results.json](https://synthcore.org/code/215/results.json)
- [Download all files as ZIP](https://synthcore.org/download/215)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.