# Besov spectral regularization for shallow ReLU

- ID: 2986
- Canonical URL: https://synthcore.org/idea/2986/besov-spectral-regularization-for-shallow-relu
- API JSON: https://synthcore.org/api/idea/2986.json
- API Markdown: https://synthcore.org/api/idea/2986.md
- Verification status: unverified
- Source: [arXiv:2609.00680](https://arxiv.org/abs/2609.00680)
- Category: regularization
- Solves: stability, generalization, accuracy
- ML areas: mlp, regularization, training-dynamics
- Math tags: functional-analysis, harmonic-analysis, approximation-theory
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Add a multiscale Besov penalty to the output of a shallow ReLU^k network, targeting the smoothness threshold that the paper proves is sufficient for finite ridge-variation representation. This suppresses pathological high-frequency output while preserving low-frequency approximation, providing a principled alternative to ordinary parameter weight decay.

## Mathematical statement

The paper defines \(\sigma_k(t)=(t)_+^k\) and the normalized ridge dictionary \(\mathcal{D}=\{\sigma_k(\omega\cdot{}-b):(\omega,b)\in\mathbb{S}^{d-1}\times[-c,c]\}\), where \(\Omega\subset\mathbb{R}^d\) is bounded and Lipschitz, \(\omega\) is a unit direction, and \(b\) is a bounded offset. Its variation norm is \(\|f\|_{\mathcal{L}_1(\mathcal{D})}=\inf\{\|\mu\|_{\mathrm{TV}}:f(x)=\int\sigma_k(\omega\cdot x-b)\,d\mu(\omega,b)\}\), with \(\mu\) a signed measure and \(\|\mu\|_{\mathrm{TV}}\) its total variation. The forward embedding states \(B^s_{p,q}(\Omega)\hookrightarrow\mathcal{L}_1(\mathcal{D})\) for \(s\ge k+d/p\) when \(0<q\le1\), and for \(s>k+d/p\) when \(1<q\le\infty\). We approximate the Besov norm using a dyadic Littlewood--Paley decomposition \(f=\sum_j\Delta_jf\), where \(\Delta_j\) contains frequencies of order \(2^j\), and penalize \(R_{s,p,q}(f)=\left(\sum_j2^{jsq}\|\Delta_jf\|_{L^p}^q\right)^{1/q}\).

## Key formulas

- $$\mathcal{D}:=\{\sigma_k(\omega\cdot{}-b):(\omega,b)\in\mathbb{S}^{d-1}\times[-c,c]\},\qquad \sigma_k(t)=(t)_+^k.$$
- $$\|f\|_{\mathcal{L}_1(\mathcal{D})}:=\inf\left\{\|\mu\|_{\mathrm{TV}}:f(x)=\int_{\mathbb{S}^{d-1}\times[-c,c]}\sigma_k(\omega\cdot x-b)\,d\mu(\omega,b)\right\}.$$
- $$B^s_{p,q}(\Omega)\hookrightarrow\mathcal{L}_1(\mathcal{D})\quad\text{if }s\ge k+d/p\ (0<q\le1),\qquad s>k+d/p\ (1<q\le\infty).$$
- $$R_{s,p,q}(f):=\left(\sum_{j=0}^{J}2^{jsq}\|\Delta_jf\|_{L^p(\Omega)}^q\right)^{1/q},\qquad \mathcal{L}=\mathcal{L}_{\mathrm{task}}+\lambda R_{s,p,q}(f_\theta).$$

## Implementation notes

Integrate the penalty at the output of a shallow ReLU^k MLP, before reduction of the task loss. Assume inputs lie in a bounded box in \(\mathbb{R}^d\). At every training step, evaluate the network on a regular grid or fixed random lattice of domain points and reshape scalar outputs into a grid. Apply an FFT, multiply by precomputed annular masks \(M_j(\xi)\) selecting frequencies near \(2^j\), and inverse FFT to obtain \(\Delta_jf\). For each band compute \(r_j=2^{js}(\operatorname{mean}_x|\Delta_jf(x)|^p+\epsilon)^{1/p}\), then set \(R=(\sum_{j=0}^{J}r_j^q)^{1/q}\) or use \(R=\max_jr_j\) for \(q=\infty\). Optimize \(\mathcal{L}_{task}+\lambda R\). Start with \(p=1\), \(q=1\), and \(s\) just above \(k+d/p\); compare with a lower-smoothness penalty and weight decay. The paper supplies the embedding threshold; grid discretization, padding, frequency masks, \(\lambda\), and the cutoff \(J\) are empirical choices. First test a width-256 shallow network on 2D regression targets made from smooth functions plus localized bumps. Compare no regularization, weight decay, and the Besov penalty at matched training FLOPs. Measure test MSE, high-frequency energy, sensitivity to input noise, and error at fixed width. Success is lower test error or perturbation sensitivity at equal training loss, with a reduced high-frequency tail rather than merely slower fitting.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.