Unverified 2026

Regular-Variation Entropy Debiasing

Implementation & benchmark of arXiv:2607.27721 — Asymptotic bias of the plug-in Shannon entropy estimator under a regularly varying occupancy model

Usefulness6/10

Difficulty5/10

Novelty7/10

Regularization Accuracy Sample efficiency Generalization Loss Regularization Rl Training Probability Statistics Measure theory Approximation theory

Source paper: Asymptotic bias of the plug-in Shannon entropy estimator under a regularly varying occupancy model arXiv:2607.27721 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Correct minibatch or trajectory-based categorical entropy estimates using the paper's power-law occupancy asymptotic. The corrected estimate adds back entropy lost through unseen rare categories, with the correction magnitude inferred from the number of distinct observed categories and an estimated tail index.

Formulas

$$\vec{\nu}(x)=\nu([x,1])=x^{-\alpha}L(x^{-1}),\qquad 0<\alpha<1,$$

$$\widehat H_n=-\sum_{j:N_j>0}\frac{N_j}{n}\log\frac{N_j}{n},\qquad H(\bm p)=-\sum_{j\ge1}p_j\log p_j,$$

$$\mathbb E[\widehat H_n]-H(\bm p)\sim-C_\alpha n^{\alpha-1}L(n),\qquad C_\alpha=\alpha\int_0^\infty\left[\mathbb E_{Z\sim\operatorname{Pois}(t)}(Z\log Z)-t\log t\right]t^{-\alpha-1}\,dt,$$

$$\widehat H_{\mathrm{RV}}=\widehat H_n+\frac{C_{\widehat\alpha}}{\Gamma(1-\widehat\alpha)}\frac{K_n}{n},\quad K_n=\sum_j\mathbf1\{N_j>0\},\quad \widehat\alpha=\operatorname{clip}\left(\frac{\log(K_{2n}/K_n)}{\log 2},\epsilon,1-\epsilon\right).$$

Mathematical statement

Let $\bm p=(p_1,p_2,\ldots)$ be a countably supported categorical distribution, $\nu=\sum_j\delta_{p_j}$ its counting measure, and $\vec\nu(x)=\nu([x,1])$. The regular-variation assumption is $\vec\nu(x)=x^{-\alpha}L(x^{-1})$, where $0<\alpha<1$ and $L$ is slowly varying, meaning $L(cx)/L(x)\to1$ for every fixed $c>0$. For $n$ samples, the plug-in estimator is $\widehat H_n=-\sum_j(N_j/n)\log(N_j/n)$, where $N_j$ is the count of category $j$. The paper establishes the downward-bias asymptotic $\mathbb E[\widehat H_n]-H(\bm p)\sim-C_\alpha n^{\alpha-1}L(n)$, with $H(\bm p)=-\sum_jp_j\log p_j$ and $C_\alpha>0$ depending only on $\alpha$. Under Poissonization, an integral representation is $C_\alpha=\alpha\int_0^\infty[\mathbb E_{Z\sim\operatorname{Pois}(t)}(Z\log Z)-t\log t]t^{-\alpha-1}\,dt$. The occupancy count $K_n=\sum_j\mathbf 1\{N_j>0\}$ has asymptotic scale $\mathbb E[K_n]\sim\Gamma(1-\alpha)n^\alpha L(n)$, allowing $n^{\alpha-1}L(n)$ to be estimated by $K_n/[\Gamma(1-\alpha)n]$. The proposed estimator is $\widehat H_{\mathrm{RV}}=\widehat H_n+[C_{\widehat\alpha}/\Gamma(1-\widehat\alpha)]K_n/n$, with $\widehat\alpha$ estimated from occupancy growth.

Implementation notes

Integrate this at the point where a model currently computes entropy from sampled categorical outcomes: an RL policy-entropy bonus, a sampled language-model token entropy diagnostic, or an entropy regularizer based on minibatch counts. Given a batch of n categorical samples, construct a hash map of category counts N_j, compute K_n as the number of nonzero entries, and calculate the ordinary plug-in entropy. Maintain a rolling window or cumulative prefix so that K_{2n} is available from twice as many samples; alternatively split a doubled batch into its first n samples and the full 2n samples. Estimate alpha with the displayed log-ratio and clip it to [0.05, 0.95]. Before training, tabulate C_alpha on a grid of alpha values by numerical quadrature of the displayed Poisson integral, then evaluate C_hat by interpolation. Add C_hat/Gamma(1-alpha_hat) times K_n/n to the empirical entropy. For policy-gradient training, detach alpha_hat, K_n, and the correction coefficient from policy logits so that the occupancy estimate does not create high-variance discrete gradients; use the corrected scalar as the entropy reward or regularization target. The mathematics supplies the asymptotic form, Poisson integral, and occupancy normalization; finite-batch smoothing is empirical. First test on synthetic categorical laws p_j proportional to j^{-q} with several q values, and compare raw plug-in entropy, Miller-Madow correction, and this estimator against exact entropy. Then test PPO on a categorical policy with a large action vocabulary. Success means lower absolute entropy error at fixed batch size and, in RL, exploration performance that is less sensitive to minibatch size without destabilizing policy updates.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.