# Noise-aware Walsh-Hadamard bottleneck

- ID: 1641
- Canonical URL: https://synthcore.org/idea/1641/noise-aware-walsh-hadamard-bottleneck
- API JSON: https://synthcore.org/api/idea/1641.json
- API Markdown: https://synthcore.org/api/idea/1641.md
- Verification status: unverified
- Source: [arXiv:2608.00539](https://arxiv.org/abs/2608.00539)
- Category: architecture
- Solves: memory, stability, speedup
- ML areas: quantization, embedding, moe-routing, transformer
- Math tags: harmonic-analysis, linear-algebra, probability, information-theory
- Ratings: usefulness 5/10; difficulty 4/10; novelty 6/10

## Idea description

Insert a Walsh-Hadamard transform before a quantized categorical or activation bottleneck and assign coordinate-dependent quantization precision using the attenuation spectrum of a quaternary symmetric noise model. Coordinates corresponding to tensor-product frequencies with many nonzero indices are attenuated by higher powers of \(\delta\), so their quantization can be made coarser with little effect on the reconstructed post-noise representation. This creates a structured, fast transform codec rather than independently quantizing the original coordinates.

## Mathematical statement

Let \(p\in[0,3/4)\) be the substitution probability and \(\delta=1-4p/3>0\). Define the normalized symmetric Hadamard matrix \(H=\frac12\begin{bmatrix}1&1&1&1\\1&1&-1&-1\\1&-1&1&-1\\1&-1&-1&1\end{bmatrix}\), which satisfies \(H^T=H\) and \(H^2=I\). The single-symbol channel is \(V=H\operatorname{diag}(1,\delta,\delta,\delta)H\), and for length-\(\ell\) strings the channel is \(W=V^{\otimes\ell}\). For an input frequency vector \(F\in\mathbb{R}^{4^\ell}\), define its Fourier representation by \(G=H^{\otimes\ell}F\). The paper's key identity is that the transformed channel output is \(D^{\otimes\ell}G\), where \(D=\operatorname{diag}(1,\delta,\delta,\delta)\). Thus each coordinate \(s=(s_1,\ldots,s_\ell)\in\{0,1,2,3\}^{\ell}\) is multiplied by \(a_s=\delta^{k(s)}\), with \(k(s)=|\{i:s_i\neq0\}|\). If scalar quantization of \(g_s\) uses step \(\Delta_s\), its approximate error variance is \(\Delta_s^2/12\), and post-channel distortion is \(\sum_s a_s^2\Delta_s^2/12\). Under a fixed total-rate approximation, distortion minimization gives \(\Delta_s\propto a_s^{-1}=\delta^{-k(s)}\), subject to clipping and practical step limits. The simplex nonnegativity constraint applies when \(F\) is a probability vector; it can be omitted for ordinary real activations.

## Key formulas

- $$H=\frac12\begin{bmatrix}1&1&1&1\\1&1&-1&-1\\1&-1&1&-1\\1&-1&-1&1\end{bmatrix},\qquad V=H\operatorname{diag}(1,\delta,\delta,\delta)H,\qquad \delta=1-\frac{4p}{3}>0.$$
- $$W=V^{\otimes \ell},\qquad G=H^{\otimes\ell}F,\qquad H^{\otimes\ell}(WF)=D^{\otimes\ell}G,\quad D=\operatorname{diag}(1,\delta,\delta,\delta).$$
- $$\widehat{g}_s=\operatorname{round}\!\left(\frac{g_s}{\Delta_s}\right)\Delta_s,\qquad \widehat{y}_s=\delta^{k(s)}\widehat{g}_s,\qquad \Delta_s=\operatorname{clip}\!\left(\Delta_0\,\delta^{-k(s)},\Delta_{\min},\Delta_{\max}\right).$$
- $$\mathbb{E}\|y-\widehat y\|_2^2\approx\frac1{12}\sum_s\delta^{2k(s)}\Delta_s^2,\qquad k(s)=|\{i:s_i\neq0\}|.$$

## Implementation notes

1. Integration point: begin with a small learned compression bottleneck, a mixture-of-experts router probability vector, or the output of a categorical embedding layer. For a vector with dimension \(4^\ell\), apply an orthonormal fast Walsh-Hadamard transform, quantize in the transformed domain, optionally apply the modeled channel contraction, and invert the transform before the downstream layer. For other dimensions, pad to the nearest convenient block size or use several independent blocks. For probability vectors, use \(F=\operatorname{softmax}(z)\); for unconstrained activations, omit simplex projection. 2. Pseudocode: `F=softmax(logits)`; `G=FWHT(F)`; for every coordinate `s`, compute `k=number_nonzero_base4_digits(s)`, set `a=delta**k`, set `Delta=clip(Delta0/a,Dmin,Dmax)`, compute `Gq=round(G/Delta)*Delta`, form `Y=FWHT(a*Gq)`, and feed `Y` to the next layer. During training use a straight-through estimator for rounding or replace rounding with uniform noise `U(-Delta/2,Delta/2)`. 3. The paper supplies the exact attenuation \(a_s=\delta^{k(s)}\). Estimate \(p\), and hence \(\delta\), from measured symbol corruption or sweep fixed values. Estimate \(\Delta_0\) from activation percentiles and the target bit budget; optionally learn one global scale while keeping the relative schedule fixed. 4. First experiment: train a small Transformer on WikiText-2 with a 256- or 1024-dimensional quantized embedding or router bottleneck. Compare ordinary per-coordinate 4-bit quantization, uniform transformed quantization, and the proposed attenuation-aware schedule at equal stored bits. Also inject synthetic quaternary substitution noise with probability \(p\). Measure validation loss, corruption robustness, compression ratio, and wall-clock overhead. Success means lower loss at the same bit budget or equal loss with fewer bits, with robustness improving monotonically as injected \(p\) increases.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.