Inverse-inequality resolution control
Implementation & benchmark of arXiv:2608.06687 — Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples
Source paper: Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples arXiv:2608.06687 ⓘ · analyzed Aug 31, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Use the network-space inverse inequality to choose derivative order, collocation resolution, and feature separation jointly instead of enforcing arbitrarily high-order residuals on an under-resolved network. This creates an anti-aliasing rule: a network whose parameters are separated by \(\underline h\) cannot represent high Sobolev frequencies without a factor \(\underline h^{-(r-s)}\), so derivative penalties above the resolvable order should be disabled or accompanied by refinement.
Formulas
Mathematical statement
For \(v_n\in L_n^k(\Theta_n)\), where \(L_n^k(\Theta_n)\) is the linearized ReLU-\(k\) network space, and \(\underline h\) is the antipodal separation distance of \(\Theta_n\), the paper proves \(\|v_n\|_{\mathcal{H}^r(\mathbb{S}^d)}\lesssim\underline h^{-(r-s)}\|v_n\|_{\mathcal{H}^s(\mathbb{S}^d)}\) for \(0\le s<r<k+1/2\). Here \(\mathcal{H}^s\) is the Sobolev norm of order \(s\), \(k\) is the ReLU smoothness/order parameter, and the hidden constant is independent of \(\underline h\). Since quasi-uniform sets with \(n\) points typically satisfy \(\underline h\asymp n^{-1/d}\), the bound becomes \(\|v_n\|_{H^r}\lesssim n^{(r-s)/d}\|v_n\|_{H^s}\). In a neural PDE solver, interpret \(s\) as the derivative order reliably measured by the collocation mesh and \(r\) as the derivative order appearing in a loss or stability estimate. The growth factor flags when high-order derivative losses will amplify coefficient noise and conditioning.
Implementation notes
1. Integration point: modify the derivative-residual loss and collocation scheduler in a PINN, neural operator, or coordinate MLP. The relevant quantities are feature-parameter separation \(\underline h\), derivative order \(r\) used by the PDE residual, and a lower-order diagnostic norm of order \(s\). 2. Pseudocode: initialize feature parameters \(\Theta\); compute \(\underline h=\min_{i\ne j}\|\theta_i-\theta_j\|\) after identifying antipodal duplicates; choose the largest candidate derivative order \(r<k+1/2\) satisfying \(\underline h^{-(r-s)}\le C_{\max}\). At each training phase, estimate lower-order residual norm \(E_s\) on a validation point set, compute the bound proxy \(B_r=\underline h^{-(r-s)}E_s\), and either use derivative loss \(L_r\) only when \(B_r\) is below a threshold or refine by adding well-separated features and collocation points. A practical loss is \(L=L_s+\lambda_rL_r\) with \(\lambda_r=\min(1,\tau/B_r)\). 3. The theorem supplies the exponent \(r-s\) and admissible range \(r<k+1/2\); the constant, actual Sobolev norms, and domain transfer must be estimated empirically. For nonspherical inputs, replace \(\underline h\) by minimum hidden-weight separation or a measured feature-kernel length scale. 4. First experiment: train width-128 and width-256 coordinate ReLU MLPs on a 2D Poisson or biharmonic problem with first-, second-, and fourth-order residual penalties. Compare fixed derivative losses against the adaptive rule at equal FLOPs. Track gradient-norm variance, residual-Jacobian condition number, dense-grid PDE residual, and final solution error. Success is that the adaptive rule prevents exploding derivative gradients and achieves equal or lower dense-grid error without increasing the collocation budget. A falsifying result is no reduction in instability or no correlation between predicted factor \(\underline h^{-(r-s)}\) and measured derivative amplification.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.