Unverified 2026

Sobolev-optimal frozen analytic features

Implementation & benchmark of arXiv:2608.18520 — Sharp Sobolev Approximation on General Domains by Linearized Shallow Networks with Analytic Activations

Usefulness6/10
Difficulty4/10
Novelty6/10

Source paper: Sharp Sobolev Approximation on General Domains by Linearized Shallow Networks with Analytic Activations arXiv:2608.18520 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a trainable shallow hidden layer by a deterministic feature dictionary generated from Chebyshev-spaced scalar parameters and quasi-uniform ridge directions. Train only the output linear map, or use the frozen layer as the first stage of a larger network, thereby eliminating hidden-layer backpropagation while retaining a constructive smooth-function approximation guarantee.

Formulas

$$\Sigma_{n}^{\sigma}(\Omega)=\left\{x\mapsto\sum_{j=1}^{n}a_{j}\sigma\big(w_{j}\cdot x+b_{j}\big):w_{j}\in\mathbb{R}^{d},\;a_{j},b_{j}\in\mathbb{R}\right\}.$$
$$\|f-f_m\|_{H^{s}}\lesssim m^{-(r-s)}\|f\|_{H^{r}},\qquad f\in H^{r},\quad r>s\geq 0.$$
$$\|f-f_{n}\|_{L^{2}(\Omega)}\lesssim n^{-\frac{r}{d}}\|f\|_{H^{r}(\Omega)},\qquad f\in H^{r}(\Omega).$$
$$t_k=\cos\!\left(\frac{(2k-1)\pi}{2m}\right),\quad u_k=u_{\min}+\frac{t_k+1}{2}(u_{\max}-u_{\min}),\quad \phi_{k,\ell}(x)=\sigma\!\left(u_kv_\ell^{\mathsf T}x+b_k\right),\quad \hat A=\arg\min_A\|\Phi A-Y\|_F^2+\lambda\|A\|_F^2.$$

Mathematical statement

The paper defines a shallow network as g(x)=sum_j a_j sigma(w_j dot x+b_j), where x is in R^d, sigma is an analytic activation, w_j and b_j are fixed inner parameters, and a_j are trainable output coefficients. Its one-dimensional result states that quasi-Chebyshev parameter sets with resolution m generate a fixed feature space with H^r-to-H^s error order m^{-(r-s)} for analytic activations satisfying a quantitative non-cancellation condition on their Taylor coefficients. Here H^r and H^s are Sobolev spaces with r>s>=0. The multidimensional construction uses quasi-uniform directions and tensor-product-type parameter sets to obtain ||f-f_n||_{L^2(Omega)} less than or comparable to n^{-r/d}||f||_{H^r(Omega)}, where Omega is bounded, f belongs to H^r(Omega), d is input dimension, and n is feature count. The implementation uses Chebyshev nodes t_k=cos((2k-1)pi/(2m)), affinely mapped to scale parameters u_k, and ridge features phi_{k,l}(x)=sigma(u_k v_l^T x+b_k), with v_l approximately uniform on the unit sphere. The theorem supplies the approximation rationale; the affine node construction and ridge readout are directly implementable adaptations.

Implementation notes

Integrate this as a frozen first hidden layer for a regression MLP or as a standalone fixed-feature model. Normalize every input coordinate to [-1,1]. Choose an analytic activation with nonvanishing Taylor coefficients, for example sigma(z)=exp(alpha z) with alpha selected so that alpha times the largest projected input remains numerically safe; tanh is also analytic but should be tested for coefficient cancellation and saturation. Generate m scale values with t_k=cos((2k-1)pi/(2m)) and map them to [u_min,u_max]. Generate L quasi-uniform directions v_l on the unit sphere using a spherical Fibonacci construction in low dimensions or normalized low-discrepancy vectors in higher dimensions. Use zero biases initially, then optionally add a second Chebyshev grid of biases. Build Phi[i,(k,l)]=sigma(u_k times dot(v_l,X_i)+b_k), with no gradient through the feature parameters. Solve the output head by A=(Phi^T Phi+lambda I)^(-1)Phi^T Y, using Cholesky for moderate width or conjugate gradients for large width. For classification, replace ridge regression with a softmax linear head. Estimate the theorem's rates empirically by fitting the slope of log validation error against log feature count. First compare against a trainable one-hidden-layer MLP and Gaussian random features on synthetic Sobolev functions formed from Fourier coefficients decaying as |q|^{-(r+d/2)}. Then test MNIST or CIFAR-10. Record wall-clock training time, peak memory, accuracy or MSE, condition number of Phi, and error slopes. Success is lower training cost and a steeper or equal error-width curve than random features at equal feature count, without numerical instability.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.