# Dimension-Calibrated Bottleneck Schedule

- ID: 2895
- Canonical URL: https://synthcore.org/idea/2895/dimension-calibrated-bottleneck-schedule
- API JSON: https://synthcore.org/api/idea/2895.json
- API Markdown: https://synthcore.org/api/idea/2895.md
- Verification status: unverified
- Source: [arXiv:2608.30549](https://arxiv.org/abs/2608.30549)
- Category: memory
- Solves: sample-efficiency, scalability, memory
- ML areas: vae, quantization, memory, embedding
- Math tags: metric-geometry, approximation-theory, information-theory
- Ratings: usefulness 5/10; difficulty 6/10; novelty 7/10

## Idea description

Use the measured power-exponential covering dimension of an activation manifold to choose the growth rate of latent codebooks, prototypes, or quantization resolution. The goal is to avoid spending parameters on a representation whose attainable resolution exceeds the intrinsic covering complexity.

## Mathematical statement

The relevant object is the covering number \(\mathcal N(\mathscr C,\varepsilon)\), the smallest number of radius-\(\varepsilon\) metric balls covering the activation set \(\mathscr C\). Its lower power-exponential dimension is \(s_*\), defined by \(s_* = \liminf_{\varepsilon\downarrow0}\log_2\log_2\mathcal N(\mathscr C,\varepsilon)/\log_2(1/\varepsilon)\). When the liminf is an actual limit and \(\mathcal N(\mathscr C,\varepsilon)\approx\exp(C\varepsilon^{-s_*})\), the paper's phase transition says that resolution \(\varepsilon\) requires approximately \(\log_2\mathcal N(\mathscr C,\varepsilon)\asymp\varepsilon^{-s_*}\) bits, while rates beyond this scaling cannot represent more than a null subset under the critical measure. For a vector-quantized bottleneck with \(K\) codes, invert this relation as \(\varepsilon(K)\approx(C^{-1}\log K)^{-1/s}\), where \(s\) is an empirical estimate of \(s_*\).

## Key formulas

- $$\underline{\operatorname{dim}}_{\mathrm{M},\exp}(\mathscr C)=\liminf_{\varepsilon\downarrow0}\frac{\log_2\log_2\mathcal N(\mathscr C,\varepsilon)}{\log_2(1/\varepsilon)}=s_*.$$
- $$\mathcal N(\mathscr C,\varepsilon)\approx\exp\!\left(C\varepsilon^{-s_*}\right),\qquad \log_2\mathcal N(\mathscr C,\varepsilon)\asymp\varepsilon^{-s_*}.$$
- $$\varepsilon(K)=\left(\frac{C}{\log K}\right)^{1/s}\quad\text{or equivalently}\quad K(\varepsilon)=\exp\!\left(C\varepsilon^{-s}\right).$$

## Implementation notes

Apply this to a VQ-VAE, residual vector quantizer, product quantizer, or token bottleneck. Collect activations \(z=f_\theta(x)\) from a calibration set and estimate \(\widehat N(r)\) with greedy farthest-point covering: start from one activation, repeatedly add the point farthest from the current centers, and count centers needed for radius \(r\). Fit \(\widehat s\) and \(\widehat C\) by robust regression of \(\log\log(\widehat N(r)+e)\) against \(\log(1/r)\), using only radii where the fit is stable. Given a fixed code budget \(K\), set the target quantization error to \(\varepsilon_K=(\widehat C/\log(K+1))^{1/\widehat s}\). During training, increase codebook size only when the measured reconstruction error is below \(0.5\varepsilon_K\) for several evaluations; if it remains above \(2\varepsilon_K\), allocate more codes or increase residual stages. The mathematical scaling is taken from the covering-number phase transition; \(\widehat C\), finite-sample radius range, and the engineering thresholds are empirical. Begin with CIFAR-10 and ImageNet-100 using a VQ-VAE with 256, 512, and 1024 total codes, comparing a fixed-code baseline and a standard perplexity-triggered growth rule at equal decoder capacity. Measure reconstruction error, downstream linear-probe accuracy, active-code perplexity, total codebook memory, and training steps to reach a target distortion. Success is equal distortion with fewer codes or faster convergence, especially when the activation geometry changes during training.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.