arXiv:2607.29042 2026 Regularization 1 ideas extracted · analyzed Aug 31, 2026
The paper proves that a discrete random variable cannot remain low-entropy under both addition and multiplication: at least one of the two transformations increases entropy by a factor of 8/7, up to a logarithmic loss. This suggests an anti-collapse regularizer for discrete neural representations, codebooks, routers, or quantized activations: representations that are simultaneously simple under additive and multiplicative composition are penalized. The most practical transfer is to estimate entropies of shuffled pairwise sums and products and impose a soft lower bound on their maximum relative to the marginal representation entropy.
Apply the entropic sum-product principle to a discrete latent variable produced by a neural network. Penalize batches in which both the shuffled pairwise sum and pairwise product have low entropy relative to the latent entropy, discouraging representations that collapse into structures with little additive or multiplicative diversity.