△ Mechanism confirmed, baseline not beaten 2026

Quasi-uniform residual least squares

Implementation & benchmark of arXiv:2608.06687 — Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples

Usefulness7/10

Difficulty4/10

Novelty6/10

Training Stability Accuracy Sample efficiency Mlp Training Loss Approximation theory Functional analysis Pde Numerical analysis

Source paper: Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples arXiv:2608.06687 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Train a frozen-feature or linearized neural network by residual least squares on deterministic quasi-uniform points rather than independently sampled collocation points. The paper's norm-equivalence result predicts that, once the number of residual points is proportional to the number of active features, the empirical residual controls the continuous residual and avoids random undersampling of localized errors.

Formulas

$$u_{n,m}\in\arg\min_{v_n\in L_n^k(\Theta_n)}\frac{1}{m}\sum_{i=1}^{m}\left(f(\eta_i^*)-\mathfrak{L}_{\beta}v_n(\eta_i^*)\right)^2.$$

$$\|u-u_{n,m}\|_{\mathcal{H}^{\beta}(\mathbb{S}^{d})}\eqsim\|f-\mathfrak{L}_{\beta}u_{n,m}\|_{\mathcal{L}^{2}(\mathbb{S}^{d})}.$$

$$\|u-u_{n,m}\|\lesssim n^{-r/d}\begin{cases}\|f\|_{\mathcal{W}^{r,p}},&d/p<r\le d/2,\ p>2,\\ \|f\|_{\mathcal{H}^{r}},&r>d/2.\end{cases}$$

$$\widehat{R}(c)=\frac{1}{m}\sum_{i=1}^{m}\left(f(\eta_i)-\sum_{j=1}^{n}c_j(\mathfrak{L}_{\beta}\phi_j)(\eta_i)\right)^2,\qquad c^*=\arg\min_c\widehat{R}(c).$$

Mathematical statement

The paper approximates the solution of $\mathfrak{L}_{\beta}u=f$ in a linear space $L_n^k(\Theta_n)$ generated by $n$ linearized ReLU-$k$ features with parameters $\Theta_n=\{\theta_j^*\}_{j=1}^n\subset\mathbb{S}^d$. Given collocation points $\{\eta_i^*\}_{i=1}^m\subset\mathbb{S}^d$, it computes $u_{n,m}$ by minimizing the empirical residual. Here $\mathfrak{L}_{\beta}$ is a positive elliptic spectral multiplier of order $\beta$, $f$ is the target forcing, and $m\gtrsim n$ means that the number of collocation points is at least a constant multiple of the feature dimension. For antipodally quasi-uniform parameters and quasi-uniform collocation points, the theorem gives $\|u-u_{n,m}\|_{\mathcal{H}^{\beta}(\mathbb{S}^d)}\eqsim\|f-\mathfrak{L}_{\beta}u_{n,m}\|_{\mathcal{L}^2(\mathbb{S}^d)}$. The approximation rate is $n^{-r/d}$ under the stated Sobolev assumptions. The transferable property is discrete-continuous residual equivalence under structured sampling, not the particular spherical PDE.

Implementation notes

1. Integration point: use the final linear readout of a frozen-feature MLP, random-feature network, neural tangent linearization, or PINN whose hidden features $\phi_j(x)$ are fixed. Replace minibatch SGD on the output coefficients with a global residual least-squares solve. For a PDE, define $A_{ij}=(\mathfrak{L}_{\beta}\phi_j)(\eta_i)$ and $b_i=f(\eta_i)$; for ordinary regression use $A_{ij}=\phi_j(\eta_i)$. 2. Pseudocode: generate $n$ approximately uniform feature parameters using a spherical Fibonacci lattice, farthest-point sampling, or antipodal pairs; generate $m=2n$ to $4n$ quasi-uniform collocation points; evaluate $A$ and $b$; solve $c^*=\arg\min_c\|Ac-b\|_2^2$ with QR or truncated SVD; periodically refresh features and repeat if a nonlinear outer loop is desired. Avoid forming $A^TA$ unless adding explicit ridge stabilization. 3. The paper supplies the reason for structured points and linear oversampling: empirical residual should be equivalent to continuous $L^2$ residual when feature parameters and points are quasi-uniform and $m\gtrsim n$. In a nonspherical domain, estimate quasi-uniformity using fill distance $h_X=\sup_x\min_i\|x-\eta_i\|$ and separation $q_X=\frac12\min_{i\ne j}\|\eta_i-\eta_j\|$, and monitor $h_X/q_X$. 4. First experiment: solve 2D Poisson on a square with frozen ReLU features of width 256 and compare uniform-grid points, iid random points, and clustered random points at equal $m\in\{n,2n,4n\}$. Hold out a dense grid for the true residual and relative $L^2$/$H^1$ error. Success is lower variance across seeds, a smaller training-to-holdout residual gap, and monotone error reduction as $m/n$ increases, ideally reaching the same holdout error with fewer points than iid sampling.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a frozen-feature residual least-squares Poisson solver and compared quasi-uniform grids with iid and intentionally clustered collocation points. The mechanism manifested clearly: grid holdout/train residual ratios were 1.044, 0.968, and 1.004 for m/n=1,2,4, while iid ratios fell from 24.27 to 1.39 as oversampling increased; localized quadrature error at m/n=4 was 1.95e-9 for the grid versus 2.52e-3 for iid. This supports structured sampling and linear oversampling in the controlled toy setting, but does not establish broad neural-network superiority.

Agent confidence: 8/10

Baseline: iid: holdout residual 30.83, 16.47, 9.23 and residual gap 24.27, 4.18, 1.39 for m/n=1,2,4; holdout residual standard deviation 4.14, 6.01, 1.19.
Idea: quasi-uniform grid: holdout residual 7.60, 7.69, 7.45 and residual gap 1.044, 0.968, 1.004 for m/n=1,2,4; localized quadrature error at m=324 was 1.95e-9 versus iid 2.52e-3.

Limitations: The implementation uses smooth Gaussian frozen features rather than frozen ReLU features, solves a simple 2D Poisson problem rather than the spherical operator, and uses dense-grid L2 residuals rather than a full H1 estimate. The deterministic grid has zero across-seed variance by construction, and no large-scale neural-network SGD or FLOP comparison was tested.

How to run: python3 experiment.py && /home/maxwelhelp/main/bin/python3 make_report.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.06687) Sep 2, 2026

Significantly WORSE than the tuned baseline (+1280.4%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Quasi_uniform_bvp_1d
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 0.0242
Idea mean: 0.3339
Effect (Δ): 0.3097 (+1280.4%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±218.2%
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the custom 1D manufactured BVP was tested because the built-in registry exposed no PDE track. The implementation used a first-order operator u+du/dx rather than the paper's spherical elliptic multiplier or second-order Poisson operator, used Adam residual training rather than frozen-feature least-squares coefficient solving, and did not compare wall-clock/FLOPs or larger models.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 1, 2026 latest Failed ✗

Agent confidence: 10/10

Built a structurally matched custom 1D elliptic BVP benchmark using the shared mlp_tiny architecture. The operator sanity check passed with maximum error 4.15e-9, and trained quasi-uniform residual systems showed a mean residual holdout/train ratio of 0.280. However, independent task MSE was significantly worse than the tuned baseline: 0.3339 versus 0.02419, paired delta +0.3097 with permutation p=0.0081; therefore the idea did not transfer as a benchmark win.

Baseline: Tuned Adam value-MSE training, lr=0.01, 25 epochs, 8-seed test MSE mean 0.0241865, std 0.0299014.
Idea: Quasi-uniform midpoint-grid residual training with boundary penalty, lr=0.01 and m/n=4, 25 epochs, 8-seed test MSE mean 0.333862, std 0.0555995; paired delta +0.3096755, p=0.0081, 0/8 wins. Mechanism signature mean trained-model residual holdout/train MSE ratio 0.279568, confirmed true.

Benchmark result