# Depth-first analytic MLP scaling

- ID: 694
- Canonical URL: https://synthcore.org/idea/694/depth-first-analytic-mlp-scaling
- API JSON: https://synthcore.org/api/idea/694.json
- API Markdown: https://synthcore.org/api/idea/694.md
- Verification status: unverified
- Source: [arXiv:2607.10589](https://arxiv.org/abs/2607.10589)
- Category: architecture
- Solves: accuracy, scalability
- ML areas: mlp, architecture, training
- Math tags: approximation-theory, polynomial, numerical-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

For smooth coordinate-based regression, replace a width-heavy MLP with a deliberately narrow but deeper ReLU network and choose depth and width using the paper's analytic-function approximation law. The hypothesis is that, at fixed parameter count, increasing depth gives a larger reduction in approximation error than increasing width when the target is close to analytic.

## Mathematical statement

The paper's main constructive bound states that for an analytic function f on [-1,1]^d, there exists a ReLU network f_NN with width C(d,rho) L^{max(lambda,1)} N (log N)^d, depth C(d,rho) L^{d-lambda+1} + C(d,rho) L (log L + log log N)^2, and uniform error at most C(d,rho,M) N^{-C(d,rho)L}. Here N is a width-scale parameter, L is a depth-scale parameter, d is input dimension, lambda is in [0,d] and controls the width-depth tradeoff, rho and M describe the analytic function class, and C denotes constants depending on the indicated arguments. The lower bound inf_{f_NN in F_NN(N,L)} sup_{f in A(rho,M)} ||f-f_NN||_{L-infinity([-1,1]^d)} >= C(d,rho,M) N^{-2L} rules out a generic exp(-N L) rate and supports treating depth as the primary approximation resource for analytic targets.

## Key formulas

- $$\left|f(\boldsymbol{x})-f_{NN}(\boldsymbol{x})\right|\leq C(d,\boldsymbol{\rho},M)N^{-C(d,\boldsymbol{\rho})L}$$
- $$\operatorname{width}\leq C(d,\boldsymbol{\rho})L^{\lambda\vee 1}N(\log N)^d$$
- $$\operatorname{depth}\leq C(d,\boldsymbol{\rho})L^{d-\lambda+1}+C(d,\boldsymbol{\rho})L(\log L+\log\log N)^2$$
- $$\inf_{f_{NN}\in\mathcal{F}_{NN}(N,L)}\sup_{f\in\mathcal{A}(\boldsymbol{\rho},M)}\|f-f_{NN}\|_{L^{\infty}([-1,1]^d)}\geq C(d,\boldsymbol{\rho},M)N^{-2L}$$

## Implementation notes

Integrate this as an architecture-selection policy for coordinate MLPs, neural fields, smooth regression, or the per-node update in a graph neural network. The exact tensors modified are the hidden affine/ReLU blocks: use a width w and depth D instead of allowing a conventional width-dominant baseline to determine both. First normalize every scalar input to [-1,1]. For a parameter budget P, sweep depth candidates D in {4, 8, 16, 32, 64}, set w approximately equal to P divided by D times the hidden-layer parameter multiplier, and reject candidates with w below 16. The training loop is: build Linear(d,w), followed by ReLU, repeated D times; append Linear(w, output_dim); train every candidate with the same optimizer, learning-rate schedule, seed set, batch size, and FLOP budget; record validation error and wall-clock time. Use the paper's bound as the design prior: for apparently analytic targets, increase D before increasing w, because the theoretical error behaves like N^{-C L}. Estimate smoothness with a small Fourier or Chebyshev probe: fit coefficients c_k and compute r_m = ||c_{k>m}||_2 / ||c_{k<=m}||_2; select depth-heavy models when r_m decays rapidly. The first experiment should use 2D or 3D regression on exp(x_1+x_2), sin(x_1) exp(x_2), and smooth PDE solutions. Compare width-heavy and depth-heavy MLPs at equal parameters and equal training FLOPs. Success is lower gridwise error at equal budget, with log error decreasing more rapidly as depth increases until optimization instability dominates.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.