Nonadaptive multiscale one-bit gradient sketch
Implementation & benchmark of arXiv:2608.02538 — Interaction Is Not Necessary for Order-Optimal 1-Bit Mean Estimation
Source paper: Interaction Is Not Necessary for Order-Optimal 1-Bit Mean Estimation arXiv:2608.02538 ⓘ · analyzed Aug 31, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace communicated floating-point gradients in synchronous federated or data-parallel training with one-bit threshold queries whose thresholds are sampled publicly before gradients are observed. Use several fixed geometric amplitude scales so the same protocol handles unknown gradient means and heavy-tailed client updates without an interactive localization round. Decode each coordinate from the scale whose neighboring estimates are statistically consistent, then apply the decoded aggregate gradient in the optimizer.
Formulas
Mathematical statement
The paper studies the moment class \(\mathcal D_k(\lambda,\sigma)=\{P:\mu_P=\mathbb E_P[X]\in[-\lambda,\lambda],\ \mathbb E_P|X-\mu_P|^k\leq\sigma^k\}\), where \(k>1\) is the known moment order, \(\lambda\) bounds the mean, and \(\sigma\) bounds the absolute central \(k\)-th moment. Its accuracy requirement is \(\sup_{P\in\mathcal D_k(\lambda,\sigma)}\Pr(|\widehat\mu-\mu_P|>\epsilon)\leq\delta\). The reported nonadaptive sample complexity is \(\log(\lambda/\sigma)+(\sigma/\epsilon)^2\log(1/\delta)\) for \(k>2\), \(\log(\lambda/\sigma)+ (\sigma/\epsilon)^2\log(\sigma/\epsilon)\log(1/\delta)\) for \(k=2\), and \(\log(\lambda/\sigma)+(\sigma/\epsilon)^{k/(k-1)}\log(1/\delta)\) for \(1<k<2\). The implementable primitive is randomized threshold integration: for scale \(B>0\), draw \(U\sim\operatorname{Unif}[-B,B]\), transmit \(b=\mathbf 1\{x\geq U\}\), and reconstruct \(z=B(2b-1)\). Whenever \(x\in[-B,B]\), \(\mathbb E_U[z\mid x]=x\); generally, the reconstruction estimates \(\operatorname{clip}_B(x)\). For centered data, the clipping bias satisfies \(|\mathbb E[x-\operatorname{clip}_B(x)]|\leq \mathbb E|x|^k/B^{k-1}\). Geometric scales \(B_j=\sigma 2^j\) cover the unknown dynamic range using \(J=\lceil\log_2(\lambda/\sigma)\rceil\) scales.
Implementation notes
Integrate the method at the client-to-server gradient communication boundary, before error-feedback accumulation and before the server optimizer update. For each coordinate g[d] of a client gradient, assign the client-coordinate pair to a fixed public scale j from a schedule; generate a threshold U[d,j] from a shared pseudorandom seed, and transmit only b[d]=1{g[d]>=U[d,j]}. The server reconstructs z[d]=B_j(2b[d]-1). Maintain a separate sum and count for every coordinate and scale. Pseudocode: choose B_j=sigma_hat*2^j for j=0,...,J; set j=hash(round,client,d) mod (J+1); draw U uniformly from [-B_j,B_j]; send b; accumulate S_j[d]+=B_j*(2*b-1) and n_j[d]+=1; compute m_j[d]=S_j[d]/n_j[d]. Select the smallest scale j for which |m_j[d]-m_{j+1}[d]| is below tau_j=B_j*sqrt(2*log(2*D*(J+1)/delta)/n_j[d]); if none qualifies, use the largest scale. Feed the selected decoded vector to SGD, momentum, or AdamW, optionally with standard residual error feedback. The paper supplies the geometric scale count, threshold-unbiased clipped estimator, and moment-based clipping control. Estimate sigma_hat during a short warm-up using coordinatewise median absolute deviation or a clipped second moment; estimate tau_j from the known threshold range and sample count. First test CIFAR-10 with ResNet-18, eight simulated non-IID clients, and full participation. Compare FP32 all-reduce, signSGD, QSGD, and this sketch at equal transmitted bits. Record test accuracy versus communication bits, loss versus communication rounds, gradient cosine similarity, and saturation rates. A useful result would be 16-32x lower communication than FP32 with at most a one-percentage-point final-accuracy loss and better stability under synthetic heavy-tailed client gradients.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.