# Stable categorical-tail body

- ID: 1621
- Canonical URL: https://synthcore.org/idea/1621/stable-categorical-tail-body
- API JSON: https://synthcore.org/api/idea/1621.json
- API Markdown: https://synthcore.org/api/idea/1621.md
- Verification status: unverified
- Source: [arXiv:2608.00081](https://arxiv.org/abs/2608.00081)
- Category: regularization
- Solves: stability, generalization, accuracy
- ML areas: regularization, vae, moe-routing
- Math tags: probability, algebra, combinatorics, convex-analysis
- Ratings: usefulness 5/10; difficulty 4/10; novelty 7/10

## Idea description

Constrain categorical distributions used by a neural module to lie in the paper's body \(\mathcal{B}_k\), which imposes a lower bound on the smallest probability based on the second-largest probability. Apply the constraint to finite-group-valued latent variables or MoE routing distributions, particularly when independently predicted categorical states are combined by group addition.

## Mathematical statement

For a probability vector \(p=(p_1,\ldots,p_k)\) on \(k\geq4\) categories, let \(p_1^\downarrow\geq p_2^\downarrow\geq\cdots\geq p_k^\downarrow\) denote its sorted coordinates. Define \(\psi_k(t)=[1-t-(1-t)^k]/(k-1)\). The paper's body is \(\mathcal{B}_k=\{p:p_k^\downarrow\geq\psi_k(p_2^\downarrow)\}\), equivalently \(1-p_2^\downarrow-(k-1)p_k^\downarrow\leq(1-p_2^\downarrow)^k\). Theorem 1.1 states that if \(p,q\in\mathcal{B}_k\), then their additive group convolution \((p*q)(z)=\sum_{x\in G}p(x)q(z-x)\) also lies in \(\mathcal{B}_k\) for every finite abelian group \(G\) of order \(k\); the analogous multiplicative result holds over finite fields. We use the inequality as a differentiable barrier. For a network probability vector \(p\), define \(v(p)=\psi_k(p_2^\downarrow)-p_k^\downarrow\); valid distributions satisfy \(v(p)\leq0\), so penalize \([v(p)]_+^2\). The theorem supplies a compositional reason for applying this penalty before group-valued composition: exact convolution preserves the feasible body.

## Key formulas

- $$\psi_k(t)=\frac{1-t-(1-t)^k}{k-1}$$
- $$\mathcal{B}_k=\left\{p:\ p_k^\downarrow\geq\psi_k\left(p_2^\downarrow\right)\right\}\quad\Longleftrightarrow\quad 1-p_2^\downarrow-(k-1)p_k^\downarrow\leq(1-p_2^\downarrow)^k$$
- $$(p*q)(z)=\sum_{x\in G}p(x)q(z-x),\qquad p,q\in\mathcal{B}_k\ \Longrightarrow\ p*q\in\mathcal{B}_k$$
- $$\mathcal{L}_{\mathrm{body}}(p)=\left[\frac{1-p_2^\downarrow-(1-p_2^\downarrow)^k}{k-1}-p_k^\downarrow\right]_+^2$$

## Implementation notes

Integrate this at the probability output of a categorical latent head, discrete diffusion state, or MoE router with exactly \(k\geq4\) choices. The cleanest MVP uses the finite cyclic group \(G=\mathbb{Z}_k\): two categorical heads produce \(p=\mathrm{softmax}(a)\) and \(q=\mathrm{softmax}(b)\), and a composition layer produces \(r(z)=\sum_{x=0}^{k-1}p(x)q((z-x)\bmod k)\). Add the body penalty to the task loss for each primitive head and optionally for each intermediate distribution. Pseudocode: `p=softmax(a); q=softmax(b); r=zeros(k); for z in range(k): r[z]=sum(p[x]*q[(z-x)%k] for x in range(k)); sp=sort(p, descending=True); p2=sp[1]; pk=sp[-1]; psi=(1-p2-(1-p2)**k)/(k-1); loss_body=relu(psi-pk)**2; loss=task_loss+lambda*loss_body`. Repeat for `q` and `r` when desired. The mathematical quantities computed exactly are \(\psi_k\), the order statistics, and group convolution; no theorem constant needs estimation. Sorting is piecewise differentiable in PyTorch or JAX; use soft top-2 and soft-min only if hard sorting causes unstable gradients. First experiment: train a small MNIST VAE with a \(k=8\) categorical latent composition, comparing no regularizer, entropy regularization, and this penalty at matched reconstruction loss. Track minimum category probability, maximum-to-minimum ratio, latent mutual information, reconstruction accuracy, and variance after composing 8 or 16 independently sampled factors. For an MoE variant, measure expert utilization entropy and validation loss at equal FLOPs. Success means fewer dead categories and lower compositional collapse without sacrificing task loss.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.