# Sobolev-optimal frozen analytic features

- ID: 2529
- Canonical URL: https://synthcore.org/idea/2529/sobolev-optimal-frozen-analytic-features
- API JSON: https://synthcore.org/api/idea/2529.json
- API Markdown: https://synthcore.org/api/idea/2529.md
- Verification status: unverified
- Source: [arXiv:2608.18520](https://arxiv.org/abs/2608.18520)
- Category: architecture
- Solves: speedup, sample-efficiency, stability
- ML areas: mlp, initialization, fine-tuning, training
- Math tags: approximation-theory, functional-analysis, harmonic-analysis, numerical-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Replace a trainable shallow hidden layer by a deterministic feature dictionary generated from Chebyshev-spaced scalar parameters and quasi-uniform ridge directions. Train only the output linear map, or use the frozen layer as the first stage of a larger network, thereby eliminating hidden-layer backpropagation while retaining a constructive smooth-function approximation guarantee.

## Mathematical statement

The paper defines a shallow network as g(x)=sum_j a_j sigma(w_j dot x+b_j), where x is in R^d, sigma is an analytic activation, w_j and b_j are fixed inner parameters, and a_j are trainable output coefficients. Its one-dimensional result states that quasi-Chebyshev parameter sets with resolution m generate a fixed feature space with H^r-to-H^s error order m^{-(r-s)} for analytic activations satisfying a quantitative non-cancellation condition on their Taylor coefficients. Here H^r and H^s are Sobolev spaces with r>s>=0. The multidimensional construction uses quasi-uniform directions and tensor-product-type parameter sets to obtain ||f-f_n||_{L^2(Omega)} less than or comparable to n^{-r/d}||f||_{H^r(Omega)}, where Omega is bounded, f belongs to H^r(Omega), d is input dimension, and n is feature count. The implementation uses Chebyshev nodes t_k=cos((2k-1)pi/(2m)), affinely mapped to scale parameters u_k, and ridge features phi_{k,l}(x)=sigma(u_k v_l^T x+b_k), with v_l approximately uniform on the unit sphere. The theorem supplies the approximation rationale; the affine node construction and ridge readout are directly implementable adaptations.

## Key formulas

- $$\Sigma_{n}^{\sigma}(\Omega)=\left\{x\mapsto\sum_{j=1}^{n}a_{j}\sigma\big(w_{j}\cdot x+b_{j}\big):w_{j}\in\mathbb{R}^{d},\;a_{j},b_{j}\in\mathbb{R}\right\}.$$
- $$\|f-f_m\|_{H^{s}}\lesssim m^{-(r-s)}\|f\|_{H^{r}},\qquad f\in H^{r},\quad r>s\geq 0.$$
- $$\|f-f_{n}\|_{L^{2}(\Omega)}\lesssim n^{-\frac{r}{d}}\|f\|_{H^{r}(\Omega)},\qquad f\in H^{r}(\Omega).$$
- $$t_k=\cos\!\left(\frac{(2k-1)\pi}{2m}\right),\quad u_k=u_{\min}+\frac{t_k+1}{2}(u_{\max}-u_{\min}),\quad \phi_{k,\ell}(x)=\sigma\!\left(u_kv_\ell^{\mathsf T}x+b_k\right),\quad \hat A=\arg\min_A\|\Phi A-Y\|_F^2+\lambda\|A\|_F^2.$$

## Implementation notes

Integrate this as a frozen first hidden layer for a regression MLP or as a standalone fixed-feature model. Normalize every input coordinate to [-1,1]. Choose an analytic activation with nonvanishing Taylor coefficients, for example sigma(z)=exp(alpha z) with alpha selected so that alpha times the largest projected input remains numerically safe; tanh is also analytic but should be tested for coefficient cancellation and saturation. Generate m scale values with t_k=cos((2k-1)pi/(2m)) and map them to [u_min,u_max]. Generate L quasi-uniform directions v_l on the unit sphere using a spherical Fibonacci construction in low dimensions or normalized low-discrepancy vectors in higher dimensions. Use zero biases initially, then optionally add a second Chebyshev grid of biases. Build Phi[i,(k,l)]=sigma(u_k times dot(v_l,X_i)+b_k), with no gradient through the feature parameters. Solve the output head by A=(Phi^T Phi+lambda I)^(-1)Phi^T Y, using Cholesky for moderate width or conjugate gradients for large width. For classification, replace ridge regression with a softmax linear head. Estimate the theorem's rates empirically by fitting the slope of log validation error against log feature count. First compare against a trainable one-hidden-layer MLP and Gaussian random features on synthetic Sobolev functions formed from Fourier coefficients decaying as |q|^{-(r+d/2)}. Then test MNIST or CIFAR-10. Record wall-clock training time, peak memory, accuracy or MSE, condition number of Phi, and error slopes. Success is lower training cost and a steeper or equal error-width curve than random features at equal feature count, without numerical instability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.