Rényi entropy robustness margin
Implementation & benchmark of arXiv:2608.04947 — Sharp Continuity of Petz and Sandwiched Rényi Conditional Entropies
Source paper: Sharp Continuity of Petz and Sandwiched Rényi Conditional Entropies arXiv:2608.04947 ⓘ · analyzed Aug 31, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Add a certified perturbation margin to entropy-based losses so that the desired entropy remains valid after input augmentation, quantization, dropout, or attention noise. Instead of treating the entropy change caused by a perturbation as an uncontrolled empirical quantity, use the sharp modulus \(\Gamma_{\alpha,D}(\delta)\) to enforce a worst-case-safe entropy target.
Formulas
Mathematical statement
For orders \(\alpha\in[1/2,1)\), the paper defines \(r=\min\{d_{\mathsf A},d_{\mathsf B}\}\), effective dimension \(D=d_{\mathsf A}r\), trace distance \(T(\rho,\sigma)=\tfrac12\lVert\rho-\sigma\rVert_1\), and \(\varepsilon=\min\{\delta,1-1/D\}\). The sharp continuity modulus is \(\Gamma_{\alpha,D}(\delta)=\frac{1}{1-\alpha}\log[(1-\varepsilon)^\alpha+(D-1)^{1-\alpha}\varepsilon^\alpha]\). It bounds the absolute change of the optimized Petz and sandwiched Rényi conditional entropies whenever the two states have trace distance at most \(\delta\), and the bound saturates at \(\log D\) for \(\delta>1-1/D\). For a neural probability vector, use the classical specialization with \(D\) support bins and total-variation distance \(\delta=\tfrac12\lVert p-q\rVert_1\); the same formula provides a conservative entropy-change budget. Here \(p,q\in\Delta^{D-1}\), \(H_\alpha(p)=\frac{1}{1-\alpha}\log\sum_{i=1}^D p_i^\alpha\), and \(\Gamma\) is used as a margin around the clean entropy.
Implementation notes
Integrate this at the output-probability or attention-probability tensor during training. For each minibatch, compute a clean distribution \(p=\operatorname{softmax}(z(x))\) and a perturbed distribution \(q=\operatorname{softmax}(z(\tilde{x}))\), where \(\tilde{x}\) is produced by augmentation, activation quantization, token dropout, or attention-logit noise. Estimate \(\hat{\delta}=\tfrac12\sum_i|p_i-q_i|\); detach this value when computing the regularizer so the model cannot reduce the penalty merely by manipulating the measured radius. Set \(\varepsilon=\min\{\hat{\delta},1-1/D\}\) and calculate \(\hat{\Gamma}=\log((1-\varepsilon)^\alpha+(D-1)^{1-\alpha}\varepsilon^\alpha)/(1-\alpha)\). If the desired clean entropy floor is \(h_0\), add \(L_{\mathrm{margin}}=\operatorname{softplus}(h_0+\hat{\Gamma}-H_\alpha(p))\), where \(H_\alpha(p)=\log(\sum_i p_i^\alpha)/(1-\alpha)\). This enforces that the clean entropy is high enough to remain above \(h_0\) for perturbations whose total-variation distance is no larger than \(\hat{\delta}\). For an entropy ceiling, use \(\operatorname{softplus}(H_\alpha(p)+\hat{\Gamma}-h_1)\). The paper supplies the exact modulus and saturation rule; the implementation estimates only the empirical TV distance and chooses \(\alpha\), target entropy, and perturbation generator. Start with a 2-layer MLP on MNIST or CIFAR-10, comparing cross-entropy plus ordinary entropy regularization against the calibrated margin under Gaussian activation noise and 8-bit activation quantization. Measure clean accuracy, corrupted accuracy, entropy variance across perturbations, and whether the observed entropy change exceeds \(\Gamma\). Success is lower entropy-violation frequency and improved corruption accuracy at equal clean accuracy, with overhead limited to one additional perturbed forward pass.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.