Stable categorical-tail body
Implementation & benchmark of arXiv:2608.00081 — Value distributions for read-once polynomials on finite fields
Source paper: Value distributions for read-once polynomials on finite fields arXiv:2608.00081 ⓘ · analyzed Aug 31, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Constrain categorical distributions used by a neural module to lie in the paper's body \(\mathcal{B}_k\), which imposes a lower bound on the smallest probability based on the second-largest probability. Apply the constraint to finite-group-valued latent variables or MoE routing distributions, particularly when independently predicted categorical states are combined by group addition.
Formulas
Mathematical statement
For a probability vector \(p=(p_1,\ldots,p_k)\) on \(k\geq4\) categories, let \(p_1^\downarrow\geq p_2^\downarrow\geq\cdots\geq p_k^\downarrow\) denote its sorted coordinates. Define \(\psi_k(t)=[1-t-(1-t)^k]/(k-1)\). The paper's body is \(\mathcal{B}_k=\{p:p_k^\downarrow\geq\psi_k(p_2^\downarrow)\}\), equivalently \(1-p_2^\downarrow-(k-1)p_k^\downarrow\leq(1-p_2^\downarrow)^k\). Theorem 1.1 states that if \(p,q\in\mathcal{B}_k\), then their additive group convolution \((p*q)(z)=\sum_{x\in G}p(x)q(z-x)\) also lies in \(\mathcal{B}_k\) for every finite abelian group \(G\) of order \(k\); the analogous multiplicative result holds over finite fields. We use the inequality as a differentiable barrier. For a network probability vector \(p\), define \(v(p)=\psi_k(p_2^\downarrow)-p_k^\downarrow\); valid distributions satisfy \(v(p)\leq0\), so penalize \([v(p)]_+^2\). The theorem supplies a compositional reason for applying this penalty before group-valued composition: exact convolution preserves the feasible body.
Implementation notes
Integrate this at the probability output of a categorical latent head, discrete diffusion state, or MoE router with exactly \(k\geq4\) choices. The cleanest MVP uses the finite cyclic group \(G=\mathbb{Z}_k\): two categorical heads produce \(p=\mathrm{softmax}(a)\) and \(q=\mathrm{softmax}(b)\), and a composition layer produces \(r(z)=\sum_{x=0}^{k-1}p(x)q((z-x)\bmod k)\). Add the body penalty to the task loss for each primitive head and optionally for each intermediate distribution. Pseudocode: p=softmax(a); q=softmax(b); r=zeros(k); for z in range(k): r[z]=sum(p[x]*q[(z-x)%k] for x in range(k)); sp=sort(p, descending=True); p2=sp[1]; pk=sp[-1]; psi=(1-p2-(1-p2)k)/(k-1); loss_body=relu(psi-pk)2; loss=task_loss+lambda*loss_body. Repeat for q and r when desired. The mathematical quantities computed exactly are \(\psi_k\), the order statistics, and group convolution; no theorem constant needs estimation. Sorting is piecewise differentiable in PyTorch or JAX; use soft top-2 and soft-min only if hard sorting causes unstable gradients. First experiment: train a small MNIST VAE with a \(k=8\) categorical latent composition, comparing no regularizer, entropy regularization, and this penalty at matched reconstruction loss. Track minimum category probability, maximum-to-minimum ratio, latent mutual information, reconstruction accuracy, and variance after composing 8 or 16 independently sampled factors. For an MoE variant, measure expert utilization entropy and validation loss at equal FLOPs. Success means fewer dead categories and lower compositional collapse without sacrificing task loss.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.