Besov spectral regularization for shallow ReLU

Integrate the penalty at the output of a shallow ReLU^k MLP, before reduction of the task loss. Assume inputs lie in a bounded box in \(\mathbb{R}^d\). At every training step, evaluate the network on a regular grid or fixed random lattice of domain points and reshape scalar outputs into a grid. Apply an FFT, multiply by precomputed annular masks \(M_j(\xi)\) selecting frequencies near \(2^j\), and inverse FFT to obtain \(\Delta_jf\). For each band compute \(r_j=2^{js}(\operatorname{mean}_x|\Delta_jf(x)|^p+\epsilon)^{1/p}\), then set \(R=(\sum_{j=0}^{J}r_j^q)^{1/q}\) or use \(R=\max_jr_j\) for \(q=\infty\). Optimize \(\mathcal{L}_{task}+\lambda R\). Start with \(p=1\), \(q=1\), and \(s\) just above \(k+d/p\); compare with a lower-smoothness penalty and weight decay. The paper supplies the embedding threshold; grid discretization, padding, frequency masks, \(\lambda\), and the cutoff \(J\) are empirical choices. First test a width-256 shallow network on 2D regression targets made from smooth functions plus localized bumps. Compare no regularization, weight decay, and the Besov penalty at matched training FLOPs. Measure test MSE, high-frequency energy, sensitivity to input noise, and error at fixed width. Success is lower test error or perturbation sensitivity at equal training loss, with a reduced high-frequency tail rather than merely slower fitting.