# Lower-bound-guided binary latent bottlenecks

- ID: 492
- Canonical URL: https://synthcore.org/idea/492/lower-bound-guided-binary-latent-bottlenecks
- API JSON: https://synthcore.org/api/idea/492.json
- API Markdown: https://synthcore.org/api/idea/492.md
- Verification status: unverified
- Source: [arXiv:2607.06750](https://arxiv.org/abs/2607.06750)
- Category: memory
- Solves: memory, accuracy, scalability
- ML areas: quantization, vae, embedding, inference-speedup
- Math tags: information-theory, approximation-theory, linear-algebra, probability
- Ratings: usefulness 5/10; difficulty 4/10; novelty 7/10

## Idea description

Use the paper's one-bit compressed-sensing lower bound to choose the number of binary latent measurements and to set a nonzero achievable-error floor during training. A sign bottleneck should not be given an unrealistically small bit budget: for approximately sparse latents, the target reconstruction error scales no faster than a power of effective sparsity divided by the number of sign measurements.

## Mathematical statement

The measurement model is \(\mathbf{y}=\operatorname{sign}(\mathbf{A}\mathbf{x})\), where \(\mathbf{x}\in\mathbb{R}^n\) is the latent vector, \(\mathbf{A}\in\mathbb{R}^{m\times n}\) is a known sub-Gaussian measurement matrix, and \(\operatorname{sign}(a)=\mathbbm{1}(a\geq0)-\mathbbm{1}(a<0)\). The signal class is approximately sparse, represented by a properly scaled \(\ell_q\) ball with effective sparsity parameter \(k\) and \(q\in[0,1]\); vectors are normalized to \(\|\mathbf{x}\|_2=1\) because sign measurements discard scale. The paper gives the uniform Euclidean lower bound \(\sup_{\mathbf{x}}\|\hat{\mathbf{x}}-\mathbf{x}\|_2\geq\widetilde{\Omega}\!\left((k/m)^{\frac{2-q}{2+q}}\right)\), where \(m\) is the number of one-bit measurements and \(\widetilde{\Omega}\) hides logarithmic factors. Special cases are \(\widetilde{\Omega}((k/m)^{1/3})\) for \(q=1\) and \(\widetilde{\Omega}(k/m)\) for \(q=0\). Inverting the scaling gives the engineering rule \(m\gtrsim k\,\varepsilon^{-(2+q)/(2-q)}\), up to logarithmic factors, to target Euclidean error \(\varepsilon\).

## Key formulas

- $$\mathbf{y}=\operatorname{sign}(\mathbf{A}\mathbf{x}),\qquad \mathbf{A}\in\mathbb{R}^{m\times n},\quad \|\mathbf{x}\|_2=1.$$
- $$\sup_{\mathbf{x}\in\mathcal{X}_{q,k}}\|\hat{\mathbf{x}}-\mathbf{x}\|_2\geq\widetilde{\Omega}\!\left(\left(\frac{k}{m}\right)^{\frac{2-q}{2+q}}\right),\qquad q\in[0,1].$$
- $$m\gtrsim k\,\varepsilon^{-\frac{2+q}{2-q}}\quad\text{(up to logarithmic factors)},$$
- $$\mathcal{L}_{\mathrm{floor}}=\max\!\left(0,\ \widehat{\mathcal{L}}_{\mathrm{rec}}-C\left(\frac{k}{m}\right)^{\frac{2-q}{2+q}}\right)$$

## Implementation notes

Integrate this into a VAE or autoencoder at the latent interface. Let the encoder produce a normalized latent z in R^n, estimate its effective sparsity k, compute binary measurements b=sign(Az+xi), and let the decoder reconstruct from b; A is a fixed Gaussian or sub-Gaussian matrix and xi is optional uniform dither. First normalize z <- z/(||z||_2+1e-8), since the mathematical model cannot recover latent norm from signs. Estimate k on each minibatch using either ||z||_1^2/||z||_2^2 as an effective-sparsity proxy or the average top-k energy; keep the estimator fixed for the first experiment. For each batch compute the theoretical floor r=C(k/m)^((2-q)/(2+q)), with C treated as a tunable calibration constant, and optimize reconstruction loss plus a hinge penalty lambda*max(0,r-L_rec) only when an overconfident target is being enforced. More importantly, compare several m values chosen by the inverted rule m proportional to k*epsilon^(-(2+q)/(2-q)) rather than arbitrary widths. Pseudocode: z=normalize(encoder(x)); k=estimate_effective_sparsity(z); b=sign(A@z+dither); xhat=decoder(b); r=C*(k/m)**((2-q)/(2+q)); loss=reconstruction(xhat,x)+beta*max(0,r-reconstruction(xhat,x)); update encoder and decoder, keeping A fixed. The theorem supplies the exponent; C, dither amplitude, and the empirical k estimator must be fitted on validation data. Start with MNIST or CIFAR-10, a small convolutional autoencoder, Gaussian A, and compare learned dense one-bit bottlenecks against this budget rule at equal stored bits and equal decoder FLOPs. Success is a predictable reconstruction-error curve with slope close to m^(-(2-q)/(2+q)), and either lower memory at matched error or lower error at matched bit budget.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.