Legendre-polynomial feature trunk
Implementation & benchmark of arXiv:2607.10589 — Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width
Source paper: Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width arXiv:2607.10589 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Construct a reusable ReLU trunk that emits approximate univariate powers or Legendre-polynomial features for each input coordinate, then combine them with a linear or low-rank polynomial head. This gives a compact explicit basis for smooth functions and can replace a large generic MLP in low-dimensional scientific regression or serve as a frozen or partially trainable front-end.
Formulas
Mathematical statement
Lemma 11 constructs a ReLU network f_{NN,1 to nu}^{(Lgd)} from R to R^nu whose i-th output approximates the degree-i Legendre polynomial L_i(x) on [-1,1]. Its width is 2^{ceil(log_2 nu)} times 10N plus 2, its depth is 5 times ceil(log_2 nu)^2 times L, and its coordinatewise error is at most (15/2) times 16^nu times N^{-5L}. The proof uses the intermediate power-feature guarantee |psi_j^{(i)}(x)-x^j| <= 30 times 4^{i-1} times N^{-5L} for j in [2^i]. For a d-dimensional input, use tensor-product basis functions prod_{r=1}^d L_{nu_r}(x_r), with L_0(x)=1, and learn coefficients over a truncated multi-index set Lambda. Here nu is the maximum univariate degree, N controls the approximation width, L controls repeated approximation depth, d is input dimension, and psi_j^{(i)} is the approximate power feature produced at construction stage i.
Implementation notes
Insert the module immediately after input normalization and before the learned prediction head. For each coordinate x_r, pass it through a shared or separately parameterized ReLU polynomial trunk producing phi_r = [1, L_1(x_r), ..., L_nu(x_r)]. For an MVP, train this trunk against analytically computed Legendre targets rather than reproducing every constant in the paper's construction. Select a truncated multi-index set Lambda, for example all nu-vectors satisfying ||nu||_1 <= K, to avoid exponential tensor-product growth. The forward pass is: normalize x to [-1,1]; for each coordinate r compute phi_r = ReLUTrunk_r(x[:,r]); for each multi-index nu in Lambda compute z_nu = product over r of phi_r[:, nu_r]; predict y = Linear(z). During pretraining, minimize sum over r and j of squared error between phi_{r,j} and L_j(x_r). During task training, either freeze the trunk, fine-tune it with a learning rate 10 times smaller than the head, or add the basis loss with coefficient alpha. The paper predicts feature error of order 16^nu N^{-5L}; measure this directly on a dense validation grid and check whether increasing L gives exponential error reduction. The first experiment should use d=2 or 3, nu=8 or 16, and targets exp(x_1+x_2), sin(x_1 x_2), and a smooth elliptic-PDE solution. Compare against a parameter-matched ReLU MLP and a fixed analytic Legendre-feature baseline. Success is lower test and gridwise error at equal parameters, or the same error with fewer hidden parameters and faster convergence.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.