Unverified 2026

Depth-first analytic MLP scaling

Implementation & benchmark of arXiv:2607.10589 — Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

Usefulness6/10
Difficulty4/10
Novelty7/10

Source paper: Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width arXiv:2607.10589 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

For smooth coordinate-based regression, replace a width-heavy MLP with a deliberately narrow but deeper ReLU network and choose depth and width using the paper's analytic-function approximation law. The hypothesis is that, at fixed parameter count, increasing depth gives a larger reduction in approximation error than increasing width when the target is close to analytic.

Formulas

$$\left|f(\boldsymbol{x})-f_{NN}(\boldsymbol{x})\right|\leq C(d,\boldsymbol{\rho},M)N^{-C(d,\boldsymbol{\rho})L}$$
$$\operatorname{width}\leq C(d,\boldsymbol{\rho})L^{\lambda\vee 1}N(\log N)^d$$
$$\operatorname{depth}\leq C(d,\boldsymbol{\rho})L^{d-\lambda+1}+C(d,\boldsymbol{\rho})L(\log L+\log\log N)^2$$
$$\inf_{f_{NN}\in\mathcal{F}_{NN}(N,L)}\sup_{f\in\mathcal{A}(\boldsymbol{\rho},M)}\|f-f_{NN}\|_{L^{\infty}([-1,1]^d)}\geq C(d,\boldsymbol{\rho},M)N^{-2L}$$

Mathematical statement

The paper's main constructive bound states that for an analytic function f on [-1,1]^d, there exists a ReLU network f_NN with width C(d,rho) L^{max(lambda,1)} N (log N)^d, depth C(d,rho) L^{d-lambda+1} + C(d,rho) L (log L + log log N)^2, and uniform error at most C(d,rho,M) N^{-C(d,rho)L}. Here N is a width-scale parameter, L is a depth-scale parameter, d is input dimension, lambda is in [0,d] and controls the width-depth tradeoff, rho and M describe the analytic function class, and C denotes constants depending on the indicated arguments. The lower bound inf_{f_NN in F_NN(N,L)} sup_{f in A(rho,M)} ||f-f_NN||_{L-infinity([-1,1]^d)} >= C(d,rho,M) N^{-2L} rules out a generic exp(-N L) rate and supports treating depth as the primary approximation resource for analytic targets.

Implementation notes

Integrate this as an architecture-selection policy for coordinate MLPs, neural fields, smooth regression, or the per-node update in a graph neural network. The exact tensors modified are the hidden affine/ReLU blocks: use a width w and depth D instead of allowing a conventional width-dominant baseline to determine both. First normalize every scalar input to [-1,1]. For a parameter budget P, sweep depth candidates D in {4, 8, 16, 32, 64}, set w approximately equal to P divided by D times the hidden-layer parameter multiplier, and reject candidates with w below 16. The training loop is: build Linear(d,w), followed by ReLU, repeated D times; append Linear(w, output_dim); train every candidate with the same optimizer, learning-rate schedule, seed set, batch size, and FLOP budget; record validation error and wall-clock time. Use the paper's bound as the design prior: for apparently analytic targets, increase D before increasing w, because the theoretical error behaves like N^{-C L}. Estimate smoothness with a small Fourier or Chebyshev probe: fit coefficients c_k and compute r_m = ||c_{k>m}||_2 / ||c_{k<=m}||_2; select depth-heavy models when r_m decays rapidly. The first experiment should use 2D or 3D regression on exp(x_1+x_2), sin(x_1) exp(x_2), and smooth PDE solutions. Compare width-heavy and depth-heavy MLPs at equal parameters and equal training FLOPs. Success is lower gridwise error at equal budget, with log error decreasing more rapidly as depth increases until optimization instability dominates.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.