# Sum-Product Anti-Collapse Regularizer

- ID: 1574
- Canonical URL: https://synthcore.org/idea/1574/sum-product-anti-collapse-regularizer
- API JSON: https://synthcore.org/api/idea/1574.json
- API Markdown: https://synthcore.org/api/idea/1574.md
- Verification status: unverified
- Source: [arXiv:2607.29042](https://arxiv.org/abs/2607.29042)
- Category: regularization
- Solves: generalization, stability, accuracy
- ML areas: embedding, regularization, quantization, moe-routing
- Math tags: information-theory, combinatorics, probability
- Ratings: usefulness 5/10; difficulty 4/10; novelty 8/10

## Idea description

Apply the entropic sum-product principle to a discrete latent variable produced by a neural network. Penalize batches in which both the shuffled pairwise sum and pairwise product have low entropy relative to the latent entropy, discouraging representations that collapse into structures with little additive or multiplicative diversity.

## Mathematical statement

The paper considers independent identically distributed discrete real-valued random variables X and X' with finite Shannon entropy H(X)=-sum_x p_X(x) log p_X(x), and proves max{H(X+X'),H(XX')} >= (8/7)H(X)-O(log H(X)). Here X+X' and XX' are obtained by applying addition and multiplication to independent draws from the same distribution. The transferable property is that both algebraic operations cannot simultaneously preserve low support complexity. For a neural discrete code Z, estimate H(Z), H(Z+Z'), and H(ZZ') from independently shuffled samples Z and Z'. Because the theorem's hidden O(log H(Z)) constant is not available for direct training, use the empirical target max(H_+,H_times) >= rho H_Z - tau, where H_+=H(Z+Z'), H_times=H(ZZ'), rho is a tunable target in [1,8/7], and tau is slack. Approximate the maximum with beta^{-1} log(exp(beta H_+)+exp(beta H_times)).

## Key formulas

- $$\max\left\{H(X+X^{\prime}),\,H(XX^{\prime})\right\}\geq\frac{8}{7}H(X)-O(\log H(X))$$
- $$H(X)=-\sum_x p_X(x)\log p_X(x),\qquad H_+=H(Z+Z'),\qquad H_{\times}=H(ZZ')$$
- $$\mathcal{L}_{\mathrm{SP}}=\left[\rho H_Z-\tau-\frac{1}{\beta}\log\left(e^{\beta H_+}+e^{\beta H_{\times}}\right)\right]_+^2$$
- $$\mathcal{L}=\mathcal{L}_{\mathrm{task}}+\lambda\mathcal{L}_{\mathrm{SP}}+\lambda_Z\,[H_{\min}-H_Z]_+^2$$

## Implementation notes

Integrate this as an auxiliary loss on a discrete representation, rather than on unrestricted floating-point activations. Suitable integration points are a VQ-VAE code index, a product-quantized embedding index, a scalar quantized bottleneck, or a discrete MoE router assignment. Let a minibatch produce integer codes z_1,...,z_B in {0,...,K-1}. Generate an independent pairing by randomly permuting the batch, z'_i=z_{pi(i)}. Map indices to centered signed values c(z)=z-(K-1)/2, optionally clip them to [-C,C], then compute s_i=c(z_i)+c(z'_i) and p_i=c(z_i)c(z'_i). Use differentiable soft histograms. For every scalar v and bin center b, define q_b(v)=exp(-(v-b)^2/(2 sigma^2))/sum_{b'} exp(-(v-b')^2/(2 sigma^2)). Average q over the batch to obtain distributions q_Z, q_+, and q_times, and calculate H(q)=-sum_b q_b log(q_b+epsilon). The pseudocode is: `z=encoder(x); z=quantize(z); z2=shuffle(z); c=center(z); c2=center(z2); qz=soft_hist(c); qp=soft_hist(c+c2); qm=soft_hist(c*c2); HZ=entropy(qz); Hp=entropy(qp); Hm=entropy(qm); hmax=logsumexp(beta*stack([Hp,Hm]))/beta; Lsp=relu(rho*HZ-tau-hmax)^2; loss=Ltask+lambda*Lsp`. Backpropagate through the soft histograms and use a straight-through estimator for hard quantization. The theorem supplies the 8/7 benchmark; binning, sigma, tau, and lambda are engineering choices. Begin with a 4-layer VQ-VAE on MNIST or CIFAR-10, K=64, and compare against standard code-usage entropy plus commitment losses. Sweep rho in {1.0,1.05,1.1,1.14}. Measure reconstruction or classification quality, dead-code count, H_Z, and max(H_+,H_times)/H_Z. Success requires improved quality or fewer dead codes at fixed codebook size without merely maximizing H_Z. Include a random code-relabeling ablation because this regularizer intentionally depends on the algebraic organization of code values.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.