✓✓ Beats tuned baseline 2026

Fully-corrective greedy neuron growth

Implementation & benchmark of arXiv:2608.20812 — Resolution-Consistent Greedy Neural Approximation on Infinite-Dimensional Spaces

Usefulness7/10

Difficulty5/10

Novelty5/10

Optimization Sample efficiency Accuracy Speedup Mlp Optimizer Training Approximation theory Optimization Functional analysis

Source paper: Resolution-Consistent Greedy Neural Approximation on Infinite-Dimensional Spaces arXiv:2608.20812 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Train a low-width network by repeatedly selecting a normalized neuron that is maximally correlated with the current residual, then refit all output coefficients jointly. This gives a constructive alternative to random initialization of all hidden units and exposes an empirical width-versus-error curve that can guide early stopping or architecture selection.

Formulas

$$\theta_{t+1}\approx\arg\max_{\theta:\ \|w\|_2\le1,\ |b|\le1}\frac{\left|\langle r_t,\Phi_\theta\rangle\right|}{\|\Phi_\theta\|_2+\varepsilon},\qquad \Phi_\theta[i]=\sigma(\langle w,P_mx_i\rangle+b).$$

$$a^{(t+1)}=\arg\min_{a\in\mathbb R^{t+1}}\left\|y-\Phi_{1:t+1}a\right\|_2^2+\lambda\|a\|_1,\qquad r_{t+1}=y-\Phi_{1:t+1}a^{(t+1)}.$$

$$\sum_{j=1}^{t}|a_j|\le R,\qquad \|y-\Phi_{1:t}a^{(t)}\|_2^2\ \text{has finite-width approximation component }O\!\left(\frac{R^2}{t}\right).$$

$$\mathcal E(f_K)\lesssim \|f-f\circ P_m\|_{L_2(P)}^2+\frac{CR^2}{K}+\frac{RB}{\sqrt n}.$$

Mathematical statement

For training samples $(x_i,y_i)_{i=1}^n$, let $r_t\in\mathbb R^n$ be the residual after $t$ atoms. For a normalized parameter $\theta=(w,b)$, define the feature vector $\Phi_\theta\in\mathbb R^n$ by $\Phi_\theta[i]=\sigma(\langle w,P_mx_i\rangle+b)$, with $\|w\|_2\le1$ and $|b|\le1$. The greedy step selects an atom approximately maximizing normalized residual correlation, $\left|\langle r_t,\Phi_\theta\rangle\right|/\|\Phi_\theta\|_2$. Fully-corrective fitting then jointly refits all coefficients by least squares plus an $\ell_1$ penalty, or under the variation constraint $\sum_j|a_j|\le R$. The constructive approximation regime predicts a squared finite-width error of order $R^2/K$, while the complete population error also contains resolution truncation and statistical terms.

Implementation notes

Use this as a hidden-layer growth optimizer for the normalized dictionary model. Initialize with only an output bias equal to the mean target. At stage $t$, compute the residual vector $r_t$ on the full small training set or a large fixed minibatch. Parameterize candidates with unconstrained $u,c$, then set $w=u/(\|u\|_2+\varepsilon)$ and $b=\tanh(c)$ so the paper's constraints hold automatically. Run 10--50 Adam steps from several random restarts to maximize $|r_t^T\Phi_\theta|/(\|\Phi_\theta\|_2+\varepsilon)$. Add the best candidate, refit all output coefficients with a small Lasso or ridge solve, and recompute the residual. Stop at width $K$ or when validation loss fails to improve. The paper supplies the normalized dictionary and the predicted $O(R^2/K)$ finite-width behavior; candidate-search quality, $R$, and the actual decay slope are empirical. First test on one-dimensional function regression and a small Burgers operator dataset. Compare against a width-$K$ two-layer MLP trained end-to-end from random initialization, matching forward/backward evaluations. Report error versus neuron count, wall-clock time, and seeds. Success is monotone validation improvement, an approximate $1/K$ residual regime, faster attainment of a target error at low width, or lower seed variance.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a fully-corrective greedy ReLU neuron-growth optimizer with constrained atom parameterization, residual-correlation search, and joint ridge refitting. Three mechanism checks passed: normalized-score scale invariance held to about 1e-12, the orthogonalized least-squares reduction identity held to 1.1e-13 relative error, and the best correlation increased monotonically with candidate-pool size (2.09 to 3.20). The width-error curve improved monotonically, but its observed log-log slope was -3.69 rather than the predicted -1 regime; greedy growth was faster than the separately trained MLP at width 16 (10.8s vs 26.6s), while the MLP sometimes achieved lower validation error.

Agent confidence: 8/10

Baseline: End-to-end width-specific tanh MLP: validation MSE at widths 0/4/8/16 = 0.5223/0.00166/0.00522/0.000128; width-16 cumulative runtime 26.62s.
Idea: Fully-corrective greedy ReLU growth: validation MSE at widths 0/4/8/16 = 0.5223/0.04478/0.003558/0.000170; width-16 cumulative runtime 10.82s. Exact LS reduction relative error 1.06e-13; normalized scaling ratios approximately 1.0; candidate-pool best scores 2.09, 2.53, 3.06, 3.20 for pool sizes 20, 100, 500, 2000; width log-log slope -3.69.

Limitations: Only a small one-dimensional noiseless function-regression task was tested; no Burgers/operator dataset, L1-constrained refit, larger-dimensional projection, or rigorous FLOP-matched comparison was performed. The MLP comparison used one seed per width, while the greedy candidate search used a small fixed number of restarts; the observed 1/K scaling prediction was not confirmed.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.20812) Sep 2, 2026

Beats the tuned baseline by 60.4% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 28.8864
Idea mean: 11.4372
Effect (Δ): -17.4492 (−60.4%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

No valid final claim should be made from this run until the baseline is routed through bench.train_model with the exact shared mlp_tiny architecture, the idea-side model identity is corrected, and the mechanism signature records trained-model per-stage residuals and correlations.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 8/10

Implemented fully-corrective greedy normalized-ReLU neuron growth and ran 8 paired seeds on the structurally matched tabular Friedman regression track. The observed test MSE was 11.437 for the idea versus 28.886 for the baseline, with delta -17.449 and permutation p=0.0081. However, the run was not protocol-clean: the baseline used a local loop rather than bench.train_model, the shared model was mislabeled as mlp_tiny, and the mechanism signature did not measure per-stage residual behaviour from trained benchmark models; therefore the apparent win is not accepted as valid stage-2 evidence.

Baseline: mean test MSE 28.886350631713867, std 1.4237160762344352, 8 seeds; best lr=0.01
Delta mean: -17.449176989866398
Idea: mean test MSE 11.437173641847467, std 3.618257764944043, 8 seeds; all tested lrs produced the same result
Idea wins: 8
Math check ls relative error: 1.8071880735156654e-14
Math check scale ratios: [0.99999999999908 1 1.000000000000092]
Permutation p value: 0.0081

Benchmark result