Fractional Hardy deficit regularizer
Implementation & benchmark of arXiv:2608.26936 — Optimal fractional discrete Hardy inequalities on the half-line
Source paper: Optimal fractional discrete Hardy inequalities on the half-line arXiv:2608.26936 ⓘ · analyzed Aug 29, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Add a boundary-aware nonlocal regularizer to hidden-state sequences by subtracting the sharp Hardy weight from the fractional discrete-Laplacian energy. The resulting penalty is provably nonnegative on finite sequences under zero-padding at the left boundary, while its position-dependent Gamma-ratio weight concentrates protection near the sequence boundary.
Formulas
Mathematical statement
The paper proves the quadratic-form inequality $(-\Delta_{\mathbb N})^{\alpha}\geq c_{\alpha}w_{\alpha}(n)$ for every $\alpha>0$, where $(-\Delta_{\mathbb N})^{\alpha}$ is the Toeplitz compression to $\ell^2(\mathbb N)$ of the full-line fractional discrete Laplacian, $w_{\alpha}(n)=\Gamma(2n-1)/\Gamma(2n-1+2\alpha)$ is a nonnegative multiplication weight, and $c_{\alpha}=4^{\alpha}\Gamma^2(\alpha+1/2)/\pi$. For a vector-valued hidden sequence $h=(h_1,\ldots,h_T)$, apply the scalar operator independently to each feature dimension and define the Hardy deficit $\mathcal R_{\alpha}(h)=\sum_d\langle h^{(d)},(-\Delta_{\mathbb N})^{\alpha}h^{(d)}\rangle-c_{\alpha}\sum_{n=1}^{T}w_{\alpha}(n)\|h_n\|_2^2\geq0$. The Toeplitz matrix has entries $L_{ij}=\ell_{|i-j|}$, where $\ell_k=(2\pi)^{-1}\int_{-\pi}^{\pi}(2-2\cos\theta)^{\alpha}e^{ik\theta}\,d\theta$ and indices are restricted to $1\leq i,j\leq T$.
Implementation notes
Integrate the regularizer at the residual-stream output of a small causal Transformer, using a hidden tensor h of shape [batch, length, width]. Optionally subtract the per-example temporal mean so the penalty targets temporal variation rather than constant feature offsets. Select alpha from {0.5, 1.0, 1.5}. Precompute the length-T Toeplitz matrix L from the Fourier coefficients of lambda(theta)=(2-2 cos(theta))^alpha, either by numerical quadrature or an FFT circulant embedding. For each batch, compute z=h-h.mean(dim=1,keepdim=True), then E=(z*(L@z)).sum() over batch, positions, and channels. Construct positions n=1,...,T, compute w[n]=gamma(2*n-1)/gamma(2*n-1+2*alpha), and compute H=c_alpha*(w[None,:,None]*z**2).sum(). Optimize task_loss + lambda_reg*(E-H)/batch_size. The theorem supplies c_alpha and w exactly; lambda_reg, mean subtraction, and alpha are empirical choices. Begin with a 2-layer d_model=256 causal Transformer on WikiText-2 and a synthetic long-range-copy task. Compare against no regularization and ordinary first-difference energy at equal FLOPs. Track validation perplexity, gradient-norm variance, loss spikes, and degradation as context length grows. A successful result is improved long-context perplexity or fewer instability events at matched training loss, with no more than 1% short-context perplexity degradation.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.