# Minimum-entropy symmetry noise

- ID: 1569
- Canonical URL: https://synthcore.org/idea/1569/minimum-entropy-symmetry-noise
- API JSON: https://synthcore.org/api/idea/1569.json
- API Markdown: https://synthcore.org/api/idea/1569.md
- Verification status: unverified
- Source: [arXiv:2607.29020](https://arxiv.org/abs/2607.29020)
- Category: regularization
- Solves: generalization, sample-efficiency, accuracy
- ML areas: regularization, data-augmentation, embedding
- Math tags: information-theory, probability, combinatorics, optimization, algebra
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Insert an additive noise layer on a discrete latent space G, choosing the noise distribution g so that the convolved latent distribution f*g is symmetric under inversion while keeping H(g) small. For binary or nearly binary categorical latents, use the paper's explicit sparse cyclic-group construction instead of uniform augmentation, preserving symmetry with substantially less randomization.

## Mathematical statement

For a countable additive group G, f and g are probability mass functions and convolution is defined by (f*g)(x) = sum_y f(y)g(x-y). A distribution g is a symmetrizer of f when (f*g)(x) = (f*g)(-x) for every x in G. The injected randomness is Shannon entropy H(g) = -sum_x g(x) log g(x). For G = Z_k with odd k = 2l+1, the paper considers f_k(-1) = 1-p and f_k(1) = p with p greater than 1/2. It defines delta = (2p-1)/(kp-l) and g_k(0) = 1-l delta, while assigning probability delta to exactly l nonzero residues and zero to the other l residues using the stated residue pattern. This sparse construction makes f_k*g_k inversion-symmetric and has H(g_k) tending to zero as p decreases to 1/2. The ML adaptation minimizes H(g) subject to the linear symmetry constraints, whose feasible solutions often lie on the boundary of the probability simplex and therefore have sparse support.

## Key formulas

- $$H(g)=-\sum_{x\in\operatorname{supp}(g)}g(x)\log g(x)$$
- $$(f*g)(x)=\sum_{y\in G}f(y)g(x-y),\qquad (f*g)(x)=(f*g)(-x)\ \text{for every }x\in G$$
- $$f_k(-1)=1-p,\quad f_k(1)=p,\quad k=2l+1,\quad \delta=\frac{2p-1}{kp-l}=\frac{2p-1}{2lp+p-l},\quad g_k(0)=1-l\delta$$
- $$\min_{g\in\Delta_k}H(g)\quad\text{subject to}\quad \sum_{y\in\mathbb Z_k}f(y)g(x-y)=\sum_{y\in\mathbb Z_k}f(y)g(-x-y)\quad\text{for every }x\in\mathbb Z_k$$

## Implementation notes

Integrate this as a stochastic augmentation layer on a categorical latent, class index, token cluster, or graph-node label. Represent each latent state by an element of Z_k. For the exact two-point path, identify the two dominant states, map them to -1 and +1, estimate p from the minibatch histogram, choose an odd k, compute l = (k-1)/2 and delta = (2p-1)/(kp-l), then construct g with g(0) = 1-l delta and the sparse residue pattern from the paper. Sample y from g and replace latent state x by (x+y) modulo k before the downstream network. For a general categorical distribution f, form a linear constraint matrix A whose row for x contains the coefficients of sum_y f(y)g(x-y) minus sum_y f(y)g(-x-y). Optimize g on the simplex using projected gradient or Frank-Wolfe, minimizing entropy subject to A g = 0; use a small positive floor during optimization, then threshold tiny entries and renormalize. Compute f from an exponential moving average of minibatch histograms, while the symmetry constraints and entropy are exact. Start with MNIST or CIFAR-10, a k = 7 categorical bottleneck, and compare no augmentation, uniform Z_7 noise, and minimum-entropy noise at equal FLOPs. Record validation accuracy, calibration, latent reconstruction error, entropy H(g), and residual max_x absolute value of (f*g)(x) minus (f*g)(-x). Success means matching or exceeding uniform-noise accuracy with lower entropy and fewer distinct transformations per example.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.