# Khintchine anti-degeneracy regularizer

- ID: 2778
- Canonical URL: https://synthcore.org/idea/2778/khintchine-anti-degeneracy-regularizer
- API JSON: https://synthcore.org/api/idea/2778.json
- API Markdown: https://synthcore.org/api/idea/2778.md
- Verification status: unverified
- Source: [arXiv:2608.27908](https://arxiv.org/abs/2608.27908)
- Category: regularization
- Solves: accuracy, stability, generalization
- ML areas: mlp, cnn, regularization
- Math tags: probability, geometry, inequalities
- Ratings: usefulness 5/10; difficulty 3/10; novelty 8/10

## Idea description

Add a regularizer that rewards each neuron's expected absolute response to random sign perturbations, normalized by the neuron's l2 norm so ordinary weight scaling cannot trivially increase the objective. Use the paper's distance-sensitive Khintchine lower bound to penalize filters close to the two-coordinate extremal set, promoting distributed and perturbation-stable feature extraction.

## Mathematical statement

For independent Rademacher variables \(\epsilon_i\in\{-1,+1\}\), define \(K_n(\omega)=\mathbb{E}_{\epsilon}|\sum_{i=1}^n\omega_i\epsilon_i|\), where \(\omega\in\mathbb{R}^n\) is a neuron or convolutional filter. The classical bound is \(K_n(\omega)\geq\|\omega\|_2/\sqrt{2}\), with equality after \(\|\omega\|_2=1\) only for vectors having exactly two nonzero coordinates of equal magnitude, up to signs and permutation. The extracted theorem gives, for normalized \(\|\omega\|_2=1\) and distance \(\tau\) from this extremal set, \(K_n(\omega)\geq1/\sqrt{2}+g(\tau)\tau\), where \(g(t)=(\sqrt{1-t^2/4}-t)/(2\sqrt{2})\). In the first regime \(0\leq\tau\leq d_4\), the paper proves \(g(\tau)\geq g(d_4)>0.056>1/36\). For implementation, set \(u=\omega/(\|\omega\|_2+\varepsilon)\), let \(|u|_{(1)}\) and \(|u|_{(2)}\) denote the two largest absolute coordinates, and define the distance to the signed/permuted extremal set as \(\tau(u)=\sqrt{2-\sqrt{2}(|u|_{(1)}+|u|_{(2)})}\). This follows because the maximum inner product with a normalized two-sparse equal-magnitude vector is \((|u|_{(1)}+|u|_{(2)})/\sqrt{2}\). A conservative theorem-derived certificate is \(B(u)=1/\sqrt{2}+\tau(u)/36\).

## Key formulas

- $$K_n(\omega)=\mathbb{E}_{\epsilon}\left|\sum_{i=1}^{n}\omega_i\epsilon_i\right|,$$
- $$K_n(\omega)\geq\frac{1}{\sqrt{2}}\|\omega\|_2,$$
- $$K_n(\omega)\geq\frac{1}{\sqrt{2}}+g(\tau)\tau,\qquad g(t)=\frac{\sqrt{1-t^{2}/4}-t}{2\sqrt{2}},\qquad \|\omega\|_2=1,$$
- $$\tau(u)=\sqrt{2-\sqrt{2}\left(|u|_{(1)}+|u|_{(2)}\right)},\qquad B(u)=\frac{1}{\sqrt{2}}+\frac{\tau(u)}{36},\qquad u=\frac{\omega}{\|\omega\|_2+\varepsilon}.$$

## Implementation notes

Integrate this at the weight tensor of an MLP layer or CNN convolution, independently for each output channel. For every filter \(w_j\), flatten all input-coordinate dimensions into one vector, compute \(r_j=\|w_j\|_2+10^{-8}\) and \(u_j=w_j/r_j\), then find the two largest entries of \(|u_j|\). Set \(\tau_j=\sqrt{\max(0,2-\sqrt{2}(a_j+b_j))}\), where \(a_j,b_j\) are the top-two magnitudes, and compute \(B_j=1/\sqrt{2}+\tau_j/36\). During training, estimate the actual Khintchine quantity with \(S\) random sign vectors: \(\widehat K_j=S^{-1}\sum_{s=1}^{S}|u_j^T\epsilon^{(s)}|\), using \(S=4\) and resampling signs each batch. Add \(L_{anti}=\lambda\,\mathrm{mean}_j[\max(0,B_{target}-\widehat K_j)]^2\), with \(B_{target}\) chosen between \(1/\sqrt{2}\) and the empirical 75th percentile of initial \(\widehat K_j\). A cheaper deterministic variant uses \(L_{cert}=\lambda\,\mathrm{mean}_j[\max(0,B_{target}-B_j)]^2\), requiring only top-two selection. Do not normalize the forward weights and do not apply the penalty to biases. First test on CIFAR-10 with a ResNet-18 and on a two-layer MLP, comparing ordinary weight decay against the proposed penalty at matched parameter count and training schedule, with \(\lambda\in\{10^{-4},10^{-3},10^{-2}\}\). Measure clean accuracy, accuracy after independently flipping 10% of input signs, calibration, mean \(\widehat K_j\), and the fraction of filters with \(\tau_j<0.1\). Success requires improved corrupted-input accuracy or calibration at matched clean accuracy, fewer near-extremal filters, and less than 5% training overhead.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.