Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples
arXiv:2608.06687
2026
Training
2 ideas extracted · analyzed Aug 31, 2026
What the math gives to ML
The paper provides a constructive stability principle for residual least-squares training: when the trainable function belongs to a finite-dimensional linearized ReLU space, quasi-uniform deterministic collocation with only linear oversampling can make the empirical residual equivalent to the continuous residual. This transfers to frozen-feature networks, neural operators, and PINN-style solvers, where poor collocation can cause aliasing, ill-conditioned least-squares systems, and large gaps between training and true residuals. Its inverse inequality also supplies a practical relation between feature-parameter separation, model size, and the highest derivative that can be reliably represented or enforced.
Ideas from this paper
△ Mechanism confirmed, baseline not beaten
2026
Train a frozen-feature or linearized neural network by residual least squares on deterministic quasi-uniform points rather than independently sampled collocation points. The paper's norm-equivalence result predicts that, once the number of residual points is proportional to the number of active features, the empirical residual controls the continuous residual and avoids random undersampling of localized errors.
Useful7/10
Difficulty4/10
Novelty6/10
Unverified
2026
Use the network-space inverse inequality to choose derivative order, collocation resolution, and feature separation jointly instead of enforcing arbitrarily high-order residuals on an under-resolved network. This creates an anti-aliasing rule: a network whose parameters are separated by \(\underline h\) cannot represent high Sobolev frequencies without a factor \(\underline h^{-(r-s)}\), so derivative penalties above the resolvable order should be disabled or accompanied by refinement.
Useful6/10
Difficulty5/10
Novelty7/10