# Fractional Hardy deficit regularizer

- ID: 215
- Canonical URL: https://synthcore.org/idea/215/fractional-hardy-deficit-regularizer
- API JSON: https://synthcore.org/api/idea/215.json
- API Markdown: https://synthcore.org/api/idea/215.md
- Verification status: unverified
- Source: [arXiv:2608.26936](https://arxiv.org/abs/2608.26936)
- Category: regularization
- Solves: stability, generalization
- ML areas: transformer, regularization, training-dynamics
- Math tags: spectral-theory, harmonic-analysis, functional-analysis
- Ratings: usefulness 5/10; difficulty 5/10; novelty 8/10

## Idea description

Add a boundary-aware nonlocal regularizer to hidden-state sequences by subtracting the sharp Hardy weight from the fractional discrete-Laplacian energy. The resulting penalty is provably nonnegative on finite sequences under zero-padding at the left boundary, while its position-dependent Gamma-ratio weight concentrates protection near the sequence boundary.

## Mathematical statement

The paper proves the quadratic-form inequality $(-\Delta_{\mathbb N})^{\alpha}\geq c_{\alpha}w_{\alpha}(n)$ for every $\alpha>0$, where $(-\Delta_{\mathbb N})^{\alpha}$ is the Toeplitz compression to $\ell^2(\mathbb N)$ of the full-line fractional discrete Laplacian, $w_{\alpha}(n)=\Gamma(2n-1)/\Gamma(2n-1+2\alpha)$ is a nonnegative multiplication weight, and $c_{\alpha}=4^{\alpha}\Gamma^2(\alpha+1/2)/\pi$. For a vector-valued hidden sequence $h=(h_1,\ldots,h_T)$, apply the scalar operator independently to each feature dimension and define the Hardy deficit $\mathcal R_{\alpha}(h)=\sum_d\langle h^{(d)},(-\Delta_{\mathbb N})^{\alpha}h^{(d)}\rangle-c_{\alpha}\sum_{n=1}^{T}w_{\alpha}(n)\|h_n\|_2^2\geq0$. The Toeplitz matrix has entries $L_{ij}=\ell_{|i-j|}$, where $\ell_k=(2\pi)^{-1}\int_{-\pi}^{\pi}(2-2\cos\theta)^{\alpha}e^{ik\theta}\,d\theta$ and indices are restricted to $1\leq i,j\leq T$.

## Key formulas

- $$(-\Delta_{\mathbb N})^{\alpha}\geq\frac{4^{\alpha}\Gamma^{2}(\alpha+1/2)}{\pi}\,\frac{\Gamma(2\,\cdot\,-1)}{\Gamma(2\,\cdot\,-1+2\alpha)}$$
- $$w_{\alpha}(n)=\frac{\Gamma(2n-1)}{\Gamma(2n-1+2\alpha)},\qquad c_{\alpha}=\frac{4^{\alpha}\Gamma^2(\alpha+1/2)}{\pi}$$
- $$\lambda(\theta)=(2-2\cos\theta)^{\alpha},\qquad \ell_k=\frac{1}{2\pi}\int_{-\pi}^{\pi}\lambda(\theta)e^{ik\theta}\,d\theta$$
- $$\mathcal R_{\alpha}(h)=\sum_{d=1}^{D}\sum_{i,j=1}^{T}h_{i,d}L_{ij}h_{j,d}-c_{\alpha}\sum_{n=1}^{T}w_{\alpha}(n)\|h_n\|_2^2\geq0$$

## Implementation notes

Integrate the regularizer at the residual-stream output of a small causal Transformer, using a hidden tensor h of shape [batch, length, width]. Optionally subtract the per-example temporal mean so the penalty targets temporal variation rather than constant feature offsets. Select alpha from {0.5, 1.0, 1.5}. Precompute the length-T Toeplitz matrix L from the Fourier coefficients of lambda(theta)=(2-2 cos(theta))^alpha, either by numerical quadrature or an FFT circulant embedding. For each batch, compute `z=h-h.mean(dim=1,keepdim=True)`, then `E=(z*(L@z)).sum()` over batch, positions, and channels. Construct positions n=1,...,T, compute `w[n]=gamma(2*n-1)/gamma(2*n-1+2*alpha)`, and compute `H=c_alpha*(w[None,:,None]*z**2).sum()`. Optimize `task_loss + lambda_reg*(E-H)/batch_size`. The theorem supplies c_alpha and w exactly; lambda_reg, mean subtraction, and alpha are empirical choices. Begin with a 2-layer d_model=256 causal Transformer on WikiText-2 and a synthetic long-range-copy task. Compare against no regularization and ordinary first-difference energy at equal FLOPs. Track validation perplexity, gradient-norm variance, loss spikes, and degradation as context length grows. A successful result is improved long-context perplexity or fewer instability events at matched training loss, with no more than 1% short-context perplexity degradation.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.