# Regular-Variation Entropy Debiasing

- ID: 1503
- Canonical URL: https://synthcore.org/idea/1503/regular-variation-entropy-debiasing
- API JSON: https://synthcore.org/api/idea/1503.json
- API Markdown: https://synthcore.org/api/idea/1503.md
- Verification status: unverified
- Source: [arXiv:2607.27721](https://arxiv.org/abs/2607.27721)
- Category: regularization
- Solves: accuracy, sample-efficiency, generalization
- ML areas: loss, regularization, rl, training
- Math tags: probability, statistics, measure-theory, approximation-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Correct minibatch or trajectory-based categorical entropy estimates using the paper's power-law occupancy asymptotic. The corrected estimate adds back entropy lost through unseen rare categories, with the correction magnitude inferred from the number of distinct observed categories and an estimated tail index.

## Mathematical statement

Let \(\bm p=(p_1,p_2,\ldots)\) be a countably supported categorical distribution, \(\nu=\sum_j\delta_{p_j}\) its counting measure, and \(\vec\nu(x)=\nu([x,1])\). The regular-variation assumption is \(\vec\nu(x)=x^{-\alpha}L(x^{-1})\), where \(0<\alpha<1\) and \(L\) is slowly varying, meaning \(L(cx)/L(x)\to1\) for every fixed \(c>0\). For \(n\) samples, the plug-in estimator is \(\widehat H_n=-\sum_j(N_j/n)\log(N_j/n)\), where \(N_j\) is the count of category \(j\). The paper establishes the downward-bias asymptotic \(\mathbb E[\widehat H_n]-H(\bm p)\sim-C_\alpha n^{\alpha-1}L(n)\), with \(H(\bm p)=-\sum_jp_j\log p_j\) and \(C_\alpha>0\) depending only on \(\alpha\). Under Poissonization, an integral representation is \(C_\alpha=\alpha\int_0^\infty[\mathbb E_{Z\sim\operatorname{Pois}(t)}(Z\log Z)-t\log t]t^{-\alpha-1}\,dt\). The occupancy count \(K_n=\sum_j\mathbf 1\{N_j>0\}\) has asymptotic scale \(\mathbb E[K_n]\sim\Gamma(1-\alpha)n^\alpha L(n)\), allowing \(n^{\alpha-1}L(n)\) to be estimated by \(K_n/[\Gamma(1-\alpha)n]\). The proposed estimator is \(\widehat H_{\mathrm{RV}}=\widehat H_n+[C_{\widehat\alpha}/\Gamma(1-\widehat\alpha)]K_n/n\), with \(\widehat\alpha\) estimated from occupancy growth.

## Key formulas

- $$\vec{\nu}(x)=\nu([x,1])=x^{-\alpha}L(x^{-1}),\qquad 0<\alpha<1,$$
- $$\widehat H_n=-\sum_{j:N_j>0}\frac{N_j}{n}\log\frac{N_j}{n},\qquad H(\bm p)=-\sum_{j\ge1}p_j\log p_j,$$
- $$\mathbb E[\widehat H_n]-H(\bm p)\sim-C_\alpha n^{\alpha-1}L(n),\qquad C_\alpha=\alpha\int_0^\infty\left[\mathbb E_{Z\sim\operatorname{Pois}(t)}(Z\log Z)-t\log t\right]t^{-\alpha-1}\,dt,$$
- $$\widehat H_{\mathrm{RV}}=\widehat H_n+\frac{C_{\widehat\alpha}}{\Gamma(1-\widehat\alpha)}\frac{K_n}{n},\quad K_n=\sum_j\mathbf1\{N_j>0\},\quad \widehat\alpha=\operatorname{clip}\left(\frac{\log(K_{2n}/K_n)}{\log 2},\epsilon,1-\epsilon\right).$$

## Implementation notes

Integrate this at the point where a model currently computes entropy from sampled categorical outcomes: an RL policy-entropy bonus, a sampled language-model token entropy diagnostic, or an entropy regularizer based on minibatch counts. Given a batch of n categorical samples, construct a hash map of category counts N_j, compute K_n as the number of nonzero entries, and calculate the ordinary plug-in entropy. Maintain a rolling window or cumulative prefix so that K_{2n} is available from twice as many samples; alternatively split a doubled batch into its first n samples and the full 2n samples. Estimate alpha with the displayed log-ratio and clip it to [0.05, 0.95]. Before training, tabulate C_alpha on a grid of alpha values by numerical quadrature of the displayed Poisson integral, then evaluate C_hat by interpolation. Add C_hat/Gamma(1-alpha_hat) times K_n/n to the empirical entropy. For policy-gradient training, detach alpha_hat, K_n, and the correction coefficient from policy logits so that the occupancy estimate does not create high-variance discrete gradients; use the corrected scalar as the entropy reward or regularization target. The mathematics supplies the asymptotic form, Poisson integral, and occupancy normalization; finite-batch smoothing is empirical. First test on synthetic categorical laws p_j proportional to j^{-q} with several q values, and compare raw plug-in entropy, Miller-Madow correction, and this estimator against exact entropy. Then test PPO on a categorical policy with a large action vocabulary. Success means lower absolute entropy error at fixed batch size and, in RL, exploration performance that is less sensitive to minibatch size without destabilizing policy updates.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.