# Capacity-Shaped Binomial Bottleneck

- ID: 371
- Canonical URL: https://synthcore.org/idea/371/capacity-shaped-binomial-bottleneck
- API JSON: https://synthcore.org/api/idea/371.json
- API Markdown: https://synthcore.org/api/idea/371.md
- Verification status: beats_baseline
- Source: [arXiv:2607.02683](https://arxiv.org/abs/2607.02683)
- Category: regularization
- Solves: generalization, memory, accuracy
- ML areas: loss, regularization, vae, quantization
- Math tags: information-theory, probability, statistics, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Replace a continuous scalar latent or probability with a stochastic count Y generated by Y|X=x ~ Binomial(n,x), and feed Y/n to the downstream network. Regularize the aggregate count distribution toward the beta-binomial distribution induced by the arcsine input X~Beta(1/2,1/2), while maximizing the mutual information carried by the count. This creates a compact discrete representation with an analytically specified, nonuniform prior that places more mass near the extreme counts without collapsing entirely to them.

## Mathematical statement

The channel is P_{Y|X}(y|x)=\binom{n}{y}x^y(1-x)^{n-y}, with x\in[0,1], y\in\{0,\ldots,n\}, and capacity C(n)=\max_{P_X} I(X;Y). The paper identifies X_r\sim\mathrm{Beta}(1/2,1/2) as an asymptotically capacity-optimal reference input. Its induced output is beta-binomial: q_r(y)=\binom{n}{y}B(y+1/2,n-y+1/2)/B(1/2,1/2), where B is the beta function. For an encoder-produced probability x, the conditional distribution is p_x(y)=P_{Y|X}(y|x). For a minibatch B, q_B(y)=|B|^{-1}\sum_{i\in B}p_{x_i}(y), and the exact batch mutual-information estimate is I_B=H(q_B)-|B|^{-1}\sum_iH(p_{x_i}), where H(r)=-\sum_y r(y)\log r(y). Add KL(q_B\|q_r) as a prior-shaping penalty. For exact gradients through expectations, Lemma 7 gives d/dx E[f(Y)|X=x]=n E^{n-1}[f(Y+1)-f(Y)|X=x], where E^{n-1} uses Y\sim\mathrm{Binomial}(n-1,x).

## Key formulas

- $$P_{Y|X}(y|x)=\binom{n}{y}x^{y}(1-x)^{n-y},\qquad x\in[0,1],\;y\in\{0,\ldots,n\}.$$
- $$q_r(y)=\binom{n}{y}\frac{B(y+1/2,n-y+1/2)}{B(1/2,1/2)},\qquad X_r\sim\mathrm{Beta}(1/2,1/2).$$
- $$I_B=H(q_B)-\frac{1}{|B|}\sum_{i\in B}H(p_{x_i}),\quad q_B(y)=\frac{1}{|B|}\sum_{i\in B}p_{x_i}(y),\quad H(r)=-\sum_y r(y)\log r(y).$$
- $$\frac{\mathrm d}{\mathrm dx}\mathbb E[f(Y)\mid X=x]=n\,\mathbb E^{n-1}[f(Y+1)-f(Y)\mid X=x],\qquad Y\sim\mathrm{Binomial}(n-1,x).$$

## Implementation notes

Integrate the module immediately after an encoder scalar probability x_i=sigmoid(a_i), or use one module per latent coordinate in a VAE or information bottleneck. Choose n between 8 and 32 so all n+1 count probabilities can be enumerated cheaply. For each x_i, compute p_i[y]=binom(n,y)x_i^y(1-x_i)^(n-y) for y=0,...,n; the downstream network receives either a sampled count Y_i/n or, for a deterministic first experiment, the expectation E[Y_i/n]=x_i. Compute q_B[y]=mean_i p_i[y], then optimize L=L_task-lambda_MI I_B+lambda_prior KL(q_B+eps || q_r+eps), where eps prevents numerical zeroes. Precompute q_r[y] from the beta-binomial expression for the selected n. If sampled counts are used, compare a straight-through estimator with exact enumeration. The paper supplies the channel probabilities, the capacity-motivated reference distribution, the entropy mutual-information objective, and the finite-difference derivative identity; lambda_MI, lambda_prior, batch smoothing, and sampling versus expectation must be estimated empirically. First test on MNIST or CIFAR-10 with a small convolutional classifier and a 16-dimensional bottleneck. Compare against a continuous 16-dimensional bottleneck and uniform categorical quantization at matched activation storage. Record validation accuracy, calibration error, estimated I_B, count histograms, endpoint mass, and robustness to additive activation noise. A successful result is equal or higher accuracy with lower-precision or discrete activation storage, reduced latent collapse, and an aggregate count histogram close to q_r.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built an enumerated Binomial bottleneck with the beta-binomial arcsine prior, exact minibatch mutual information, KL shaping, and a discrete sampled-count deployment path. The derivative identity was reproduced with maximum absolute error 1.0e-9; on a fixed small digits split, the idea improved deterministic accuracy from 0.8756 to 0.8978 and sampled-count accuracy was 0.8956, while reducing KL to the reference prior from 0.2945 to 0.0966 and storage from 32 to 3.17 bits per coordinate. This is a promising toy signal, but not a definitive generalization win because only one small dataset/run was tested and the baseline was not a matched-bit quantized control.

### Mechanism check

- Verdict: Built an enumerated Binomial bottleneck with the beta-binomial arcsine prior, exact minibatch mutual information, KL shaping, and a discrete sampled-count deployment path. The derivative identity was reproduced with maximum absolute error 1.0e-9; on a fixed small digits split, the idea improved deterministic accuracy from 0.8756 to 0.8978 and sampled-count accuracy was 0.8956, while reducing KL to the reference prior from 0.2945 to 0.0966 and storage from 32 to 3.17 bits per coordinate. This is a promising toy signal, but not a definitive generalization win because only one small dataset/run was tested and the baseline was not a matched-bit quantized control.
- Confidence: 7/10
- Limitations: Tested only sklearn digits with one fixed train/test split and one seed. No MNIST/CIFAR experiment, repeated-seed confidence intervals, calibration, additive-noise robustness, matched-storage uniform quantization baseline, or fully stochastic straight-through training was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_binomial.py](https://synthcore.org/code/1277/bench_binomial.py)
- [bench_report.json](https://synthcore.org/code/1277/bench_report.json)
- [binomial_bottleneck_experiment.py](https://synthcore.org/code/1277/binomial_bottleneck_experiment.py)
- [report.md](https://synthcore.org/code/1277/report.md)
- [report_bench_2026-09-04T003730.md](https://synthcore.org/code/1277/report_bench_2026-09-04T003730.md)
- [results.json](https://synthcore.org/code/1277/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1277)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.