# Quasi-uniform residual least squares

- ID: 1905
- Canonical URL: https://synthcore.org/idea/1905/quasi-uniform-residual-least-squares
- API JSON: https://synthcore.org/api/idea/1905.json
- API Markdown: https://synthcore.org/api/idea/1905.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.06687](https://arxiv.org/abs/2608.06687)
- Category: training
- Solves: stability, accuracy, sample-efficiency
- ML areas: mlp, training, loss
- Math tags: approximation-theory, functional-analysis, pde, numerical-analysis
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Train a frozen-feature or linearized neural network by residual least squares on deterministic quasi-uniform points rather than independently sampled collocation points. The paper's norm-equivalence result predicts that, once the number of residual points is proportional to the number of active features, the empirical residual controls the continuous residual and avoids random undersampling of localized errors.

## Mathematical statement

The paper approximates the solution of \(\mathfrak{L}_{\beta}u=f\) in a linear space \(L_n^k(\Theta_n)\) generated by \(n\) linearized ReLU-\(k\) features with parameters \(\Theta_n=\{\theta_j^*\}_{j=1}^n\subset\mathbb{S}^d\). Given collocation points \(\{\eta_i^*\}_{i=1}^m\subset\mathbb{S}^d\), it computes \(u_{n,m}\) by minimizing the empirical residual. Here \(\mathfrak{L}_{\beta}\) is a positive elliptic spectral multiplier of order \(\beta\), \(f\) is the target forcing, and \(m\gtrsim n\) means that the number of collocation points is at least a constant multiple of the feature dimension. For antipodally quasi-uniform parameters and quasi-uniform collocation points, the theorem gives \(\|u-u_{n,m}\|_{\mathcal{H}^{\beta}(\mathbb{S}^d)}\eqsim\|f-\mathfrak{L}_{\beta}u_{n,m}\|_{\mathcal{L}^2(\mathbb{S}^d)}\). The approximation rate is \(n^{-r/d}\) under the stated Sobolev assumptions. The transferable property is discrete-continuous residual equivalence under structured sampling, not the particular spherical PDE.

## Key formulas

- $$u_{n,m}\in\arg\min_{v_n\in L_n^k(\Theta_n)}\frac{1}{m}\sum_{i=1}^{m}\left(f(\eta_i^*)-\mathfrak{L}_{\beta}v_n(\eta_i^*)\right)^2.$$
- $$\|u-u_{n,m}\|_{\mathcal{H}^{\beta}(\mathbb{S}^{d})}\eqsim\|f-\mathfrak{L}_{\beta}u_{n,m}\|_{\mathcal{L}^{2}(\mathbb{S}^{d})}.$$
- $$\|u-u_{n,m}\|\lesssim n^{-r/d}\begin{cases}\|f\|_{\mathcal{W}^{r,p}},&d/p<r\le d/2,\ p>2,\\ \|f\|_{\mathcal{H}^{r}},&r>d/2.\end{cases}$$
- $$\widehat{R}(c)=\frac{1}{m}\sum_{i=1}^{m}\left(f(\eta_i)-\sum_{j=1}^{n}c_j(\mathfrak{L}_{\beta}\phi_j)(\eta_i)\right)^2,\qquad c^*=\arg\min_c\widehat{R}(c).$$

## Implementation notes

1. Integration point: use the final linear readout of a frozen-feature MLP, random-feature network, neural tangent linearization, or PINN whose hidden features \(\phi_j(x)\) are fixed. Replace minibatch SGD on the output coefficients with a global residual least-squares solve. For a PDE, define \(A_{ij}=(\mathfrak{L}_{\beta}\phi_j)(\eta_i)\) and \(b_i=f(\eta_i)\); for ordinary regression use \(A_{ij}=\phi_j(\eta_i)\). 2. Pseudocode: generate \(n\) approximately uniform feature parameters using a spherical Fibonacci lattice, farthest-point sampling, or antipodal pairs; generate \(m=2n\) to \(4n\) quasi-uniform collocation points; evaluate \(A\) and \(b\); solve \(c^*=\arg\min_c\|Ac-b\|_2^2\) with QR or truncated SVD; periodically refresh features and repeat if a nonlinear outer loop is desired. Avoid forming \(A^TA\) unless adding explicit ridge stabilization. 3. The paper supplies the reason for structured points and linear oversampling: empirical residual should be equivalent to continuous \(L^2\) residual when feature parameters and points are quasi-uniform and \(m\gtrsim n\). In a nonspherical domain, estimate quasi-uniformity using fill distance \(h_X=\sup_x\min_i\|x-\eta_i\|\) and separation \(q_X=\frac12\min_{i\ne j}\|\eta_i-\eta_j\|\), and monitor \(h_X/q_X\). 4. First experiment: solve 2D Poisson on a square with frozen ReLU features of width 256 and compare uniform-grid points, iid random points, and clustered random points at equal \(m\in\{n,2n,4n\}\). Hold out a dense grid for the true residual and relative \(L^2\)/\(H^1\) error. Success is lower variance across seeds, a smaller training-to-holdout residual gap, and monotone error reduction as \(m/n\) increases, ideally reaching the same holdout error with fewer points than iid sampling.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a frozen-feature residual least-squares Poisson solver and compared quasi-uniform grids with iid and intentionally clustered collocation points. The mechanism manifested clearly: grid holdout/train residual ratios were 1.044, 0.968, and 1.004 for m/n=1,2,4, while iid ratios fell from 24.27 to 1.39 as oversampling increased; localized quadrature error at m/n=4 was 1.95e-9 for the grid versus 2.52e-3 for iid. This supports structured sampling and linear oversampling in the controlled toy setting, but does not establish broad neural-network superiority.

### Mechanism check

- Verdict: Built a frozen-feature residual least-squares Poisson solver and compared quasi-uniform grids with iid and intentionally clustered collocation points. The mechanism manifested clearly: grid holdout/train residual ratios were 1.044, 0.968, and 1.004 for m/n=1,2,4, while iid ratios fell from 24.27 to 1.39 as oversampling increased; localized quadrature error at m/n=4 was 1.95e-9 for the grid versus 2.52e-3 for iid. This supports structured sampling and linear oversampling in the controlled toy setting, but does not establish broad neural-network superiority.
- Confidence: 8/10
- Limitations: The implementation uses smooth Gaussian frozen features rather than frozen ReLU features, solves a simple 2D Poisson problem rather than the spherical operator, and uses dense-grid L2 residuals rather than a full H1 estimate. The deterministic grid has zero across-seed variance by construction, and no large-scale neural-network SGD or FLOP comparison was tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [audit.json](https://synthcore.org/code/724/audit.json)
- [bench_report.json](https://synthcore.org/code/724/bench_report.json)
- [experiment.py](https://synthcore.org/code/724/experiment.py)
- [make_report.py](https://synthcore.org/code/724/make_report.py)
- [report.md](https://synthcore.org/code/724/report.md)
- [report_bench_2026-09-01T012121.md](https://synthcore.org/code/724/report_bench_2026-09-01T012121.md)
- [results.json](https://synthcore.org/code/724/results.json)
- [stage2_bench.py](https://synthcore.org/code/724/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/724)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.