{
 "artifacts": null,
 "category": "architecture",
 "description": "Insert a Walsh-Hadamard transform before a quantized categorical or activation bottleneck and assign coordinate-dependent quantization precision using the attenuation spectrum of a quaternary symmetric noise model. Coordinates corresponding to tensor-product frequencies with many nonzero indices are attenuated by higher powers of \\(\\delta\\), so their quantization can be made coarser with little effect on the reconstructed post-noise representation. This creates a structured, fast transform codec rather than independently quantizing the original coordinates.",
 "formulas_latex": [
  "$$H=\\frac12\\begin{bmatrix}1\u00261\u00261\u00261\\\\1\u00261\u0026-1\u0026-1\\\\1\u0026-1\u00261\u0026-1\\\\1\u0026-1\u0026-1\u00261\\end{bmatrix},\\qquad V=H\\operatorname{diag}(1,\\delta,\\delta,\\delta)H,\\qquad \\delta=1-\\frac{4p}{3}\u003e0.$$",
  "$$W=V^{\\otimes \\ell},\\qquad G=H^{\\otimes\\ell}F,\\qquad H^{\\otimes\\ell}(WF)=D^{\\otimes\\ell}G,\\quad D=\\operatorname{diag}(1,\\delta,\\delta,\\delta).$$",
  "$$\\widehat{g}_s=\\operatorname{round}\\!\\left(\\frac{g_s}{\\Delta_s}\\right)\\Delta_s,\\qquad \\widehat{y}_s=\\delta^{k(s)}\\widehat{g}_s,\\qquad \\Delta_s=\\operatorname{clip}\\!\\left(\\Delta_0\\,\\delta^{-k(s)},\\Delta_{\\min},\\Delta_{\\max}\\right).$$",
  "$$\\mathbb{E}\\|y-\\widehat y\\|_2^2\\approx\\frac1{12}\\sum_s\\delta^{2k(s)}\\Delta_s^2,\\qquad k(s)=|\\{i:s_i\\neq0\\}|.$$"
 ],
 "id": 1641,
 "implementation": "1. Integration point: begin with a small learned compression bottleneck, a mixture-of-experts router probability vector, or the output of a categorical embedding layer. For a vector with dimension \\(4^\\ell\\), apply an orthonormal fast Walsh-Hadamard transform, quantize in the transformed domain, optionally apply the modeled channel contraction, and invert the transform before the downstream layer. For other dimensions, pad to the nearest convenient block size or use several independent blocks. For probability vectors, use \\(F=\\operatorname{softmax}(z)\\); for unconstrained activations, omit simplex projection. 2. Pseudocode: `F=softmax(logits)`; `G=FWHT(F)`; for every coordinate `s`, compute `k=number_nonzero_base4_digits(s)`, set `a=delta**k`, set `Delta=clip(Delta0/a,Dmin,Dmax)`, compute `Gq=round(G/Delta)*Delta`, form `Y=FWHT(a*Gq)`, and feed `Y` to the next layer. During training use a straight-through estimator for rounding or replace rounding with uniform noise `U(-Delta/2,Delta/2)`. 3. The paper supplies the exact attenuation \\(a_s=\\delta^{k(s)}\\). Estimate \\(p\\), and hence \\(\\delta\\), from measured symbol corruption or sweep fixed values. Estimate \\(\\Delta_0\\) from activation percentiles and the target bit budget; optionally learn one global scale while keeping the relative schedule fixed. 4. First experiment: train a small Transformer on WikiText-2 with a 256- or 1024-dimensional quantized embedding or router bottleneck. Compare ordinary per-coordinate 4-bit quantization, uniform transformed quantization, and the proposed attenuation-aware schedule at equal stored bits. Also inject synthetic quaternary substitution noise with probability \\(p\\). Measure validation loss, corruption robustness, compression ratio, and wall-clock overhead. Success means lower loss at the same bit budget or equal loss with fewer bits, with robustness improving monotonically as injected \\(p\\) increases.",
 "math_summary": "Let \\(p\\in[0,3/4)\\) be the substitution probability and \\(\\delta=1-4p/3\u003e0\\). Define the normalized symmetric Hadamard matrix \\(H=\\frac12\\begin{bmatrix}1\u00261\u00261\u00261\\\\1\u00261\u0026-1\u0026-1\\\\1\u0026-1\u00261\u0026-1\\\\1\u0026-1\u0026-1\u00261\\end{bmatrix}\\), which satisfies \\(H^T=H\\) and \\(H^2=I\\). The single-symbol channel is \\(V=H\\operatorname{diag}(1,\\delta,\\delta,\\delta)H\\), and for length-\\(\\ell\\) strings the channel is \\(W=V^{\\otimes\\ell}\\). For an input frequency vector \\(F\\in\\mathbb{R}^{4^\\ell}\\), define its Fourier representation by \\(G=H^{\\otimes\\ell}F\\). The paper's key identity is that the transformed channel output is \\(D^{\\otimes\\ell}G\\), where \\(D=\\operatorname{diag}(1,\\delta,\\delta,\\delta)\\). Thus each coordinate \\(s=(s_1,\\ldots,s_\\ell)\\in\\{0,1,2,3\\}^{\\ell}\\) is multiplied by \\(a_s=\\delta^{k(s)}\\), with \\(k(s)=|\\{i:s_i\\neq0\\}|\\). If scalar quantization of \\(g_s\\) uses step \\(\\Delta_s\\), its approximate error variance is \\(\\Delta_s^2/12\\), and post-channel distortion is \\(\\sum_s a_s^2\\Delta_s^2/12\\). Under a fixed total-rate approximation, distortion minimization gives \\(\\Delta_s\\propto a_s^{-1}=\\delta^{-k(s)}\\), subject to clipping and practical step limits. The simplex nonnegativity constraint applies when \\(F\\) is a probability vector; it can be omitted for ordinary real activations.",
 "math_tags": [
  "harmonic-analysis",
  "linear-algebra",
  "probability",
  "information-theory"
 ],
 "ml_areas": [
  "quantization",
  "embedding",
  "moe-routing",
  "transformer"
 ],
 "paper": {
  "arxiv_id": "2608.00539",
  "arxiv_url": "https://arxiv.org/abs/2608.00539",
  "summary_what_math_gives_to_ml": "The paper provides a concrete spectral representation of a quaternary symmetric noise channel: a Hadamard transform diagonalizes the channel, turning symbol substitutions into independent coordinate contractions whose strengths are powers of \\(\\delta=1-4p/3\\). This suggests a channel-aware neural representation or quantizer that allocates precision according to transform frequency rather than treating all coordinates equally. The most practical transfer is a Walsh-Hadamard bottleneck for categorical distributions, embeddings, or learned activation compression, with coarser quantization on coordinates that the modeled noise strongly attenuates and finer quantization on stable coordinates.",
  "title": "Frequency Coding over Noisy Sampling",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 6,
  "usefulness": 5
 },
 "solves": [
  "memory",
  "stability",
  "speedup"
 ],
 "title": "Noise-aware Walsh-Hadamard bottleneck",
 "url": "https://synthcore.org/idea/1641/noise-aware-walsh-hadamard-bottleneck",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}