# Polarized Gaussian bottleneck

- ID: 1997
- Canonical URL: https://synthcore.org/idea/1997/polarized-gaussian-bottleneck
- API JSON: https://synthcore.org/api/idea/1997.json
- API Markdown: https://synthcore.org/api/idea/1997.md
- Verification status: unverified
- Source: [arXiv:2608.08472](https://arxiv.org/abs/2608.08472)
- Category: regularization
- Solves: memory, generalization, accuracy
- ML areas: mlp, vae, moe-routing, regularization, quantization
- Math tags: probability, convex-analysis, spectral-theory, statistics
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Replace isotropic variance control in a bottleneck or router with a spectral polarization penalty that drives each latent direction toward either variance 0 or variance 1. The intended result is an automatically selected active subspace: inactive coordinates can be pruned or quantized aggressively, while active coordinates retain information instead of being uniformly attenuated.

## Mathematical statement

The paper's (B)-theorem states that for standard Gaussian measure \(\gamma\) and a symmetric convex set \(K\subset\mathbb{R}^n\), \(\gamma(\sqrt{ab}K)\geq\sqrt{\gamma(aK)\gamma(bK)}\) for \(a,b>0\). Equivalently, \(f(t)=\log\gamma(e^tK)\) is concave, so \(f((s+t)/2)\geq(f(s)+f(t))/2\). The stability result says that if the defect \(\epsilon\) in this inequality is small, then every eigenvalue \(\lambda_i\) of the covariance matrix of the probability measure \(\gamma(\cdot\mid K)\) satisfies either \(\lambda_i\geq 1-O(\epsilon)\) or \(\lambda_i\leq O(\epsilon)\). Here \(\lambda_i\) is the variance along the \(i\)-th principal component, and standard Gaussian variance is 1. The neural adaptation explicitly penalizes \(\lambda_i(1-\lambda_i)\), whose minima are exactly the theorem's two spectral endpoints; this is a surrogate for the paper's near-optimizer dichotomy and is not claimed to hold without the Gaussian-convex assumptions.

## Key formulas

- $$\gamma(\sqrt{ab}K)\geq\sqrt{\gamma(aK)\gamma(bK)},\qquad a,b>0,$$
- $$f(t)=\log\gamma(e^tK),\qquad f\!\left(\frac{s+t}{2}\right)\geq\frac{f(s)+f(t)}{2},$$
- $$\lambda_i\geq 1-O(\epsilon)\quad\text{or}\quad\lambda_i\leq O(\epsilon),$$
- $$\mathcal{L}_{\mathrm{pol}}=\frac{1}{d}\sum_{i=1}^{d}\lambda_i(1-\lambda_i),\qquad \lambda_i=\operatorname{eig}_i\!\left(\frac{1}{B-1}(Z-\bar z\mathbf{1}^{\mathsf T})(Z-\bar z\mathbf{1}^{\mathsf T})^{\mathsf T}\right).$$

## Implementation notes

Integrate this at the output of a stochastic bottleneck, VAE latent block, or MoE router before the next linear layer. Let \(Z\in\mathbb{R}^{d\times B}\) contain one centered latent vector per batch item, \(\bar z\) be the batch mean, and \(C=(B-1)^{-1}(Z-\bar z\mathbf{1}^{\mathsf T})(Z-\bar z\mathbf{1}^{\mathsf T})^{\mathsf T}\). To make the Gaussian scale meaningful, first normalize each latent coordinate or apply a learned diagonal scale constrained to \([0,1]\), then compute the symmetric eigendecomposition \(C=U\operatorname{diag}(\lambda_1,\ldots,\lambda_d)U^{\mathsf T}\). Add \(\beta d^{-1}\sum_i\lambda_i(1-\lambda_i)\) to the task loss, optionally with a variance-budget term \(\alpha(\sum_i\lambda_i-r)^2\) targeting effective rank \(r\). Pseudocode: `z = bottleneck(x); zc = z - mean(z, batch); C = zc.T @ zc / (B-1); lam = eigvalsh(C); loss = task_loss + beta*mean(lam*(1-lam)) + alpha*(lam.sum()-r)**2; loss.backward()`. For large \(d\), estimate only the extreme eigenvalues with 5-10 Lanczos iterations, or use a Hutchinson estimator for the trace of \(C-C^2\). The paper supplies the Gaussian-convex spectral target and endpoint interpretation; covariance estimation, unit-variance normalization, and the rank budget are engineering choices. First test on MNIST or CIFAR-10 with a 64-to-16 autoencoder bottleneck, comparing no regularizer, ordinary \(\|C-I\|_F^2\), and the polarization loss at equal parameter count. Measure reconstruction or classification accuracy, eigenvalue histograms, the number of eigenvalues below 0.05 or above 0.95, and post-training pruning of low-variance coordinates. Success means a bimodal spectrum with no accuracy loss and at least 2x removable bottleneck coordinates, or improved accuracy at the same effective rank. A second test can apply the loss to router assignment embeddings and measure expert load balance and routing entropy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.