Unverified 2026

Inverse-inequality resolution control

Implementation & benchmark of arXiv:2608.06687 — Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples

Usefulness6/10
Difficulty5/10
Novelty7/10

Source paper: Optimal Neural Network Approximation via Empirical Least Squares with Deterministic Samples arXiv:2608.06687 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Use the network-space inverse inequality to choose derivative order, collocation resolution, and feature separation jointly instead of enforcing arbitrarily high-order residuals on an under-resolved network. This creates an anti-aliasing rule: a network whose parameters are separated by \(\underline h\) cannot represent high Sobolev frequencies without a factor \(\underline h^{-(r-s)}\), so derivative penalties above the resolvable order should be disabled or accompanied by refinement.

Formulas

$$\|v_{n}\|_{\mathcal{H}^{r}(\mathbb{S}^{d})}\lesssim\underline{h}^{-(r-s)}\|v_{n}\|_{\mathcal{H}^{s}(\mathbb{S}^{d})},\qquad 0\le s<r<k+\tfrac{1}{2}.$$
$$\underline h\asymp n^{-1/d}\quad\Longrightarrow\quad\|v_n\|_{\mathcal{H}^{r}}\lesssim n^{(r-s)/d}\|v_n\|_{\mathcal{H}^{s}}.$$
$$\lambda_r=\min\left(1,\frac{\tau}{\underline h^{-(r-s)}E_s}\right),\qquad L=L_s+\lambda_rL_r.$$

Mathematical statement

For \(v_n\in L_n^k(\Theta_n)\), where \(L_n^k(\Theta_n)\) is the linearized ReLU-\(k\) network space, and \(\underline h\) is the antipodal separation distance of \(\Theta_n\), the paper proves \(\|v_n\|_{\mathcal{H}^r(\mathbb{S}^d)}\lesssim\underline h^{-(r-s)}\|v_n\|_{\mathcal{H}^s(\mathbb{S}^d)}\) for \(0\le s<r<k+1/2\). Here \(\mathcal{H}^s\) is the Sobolev norm of order \(s\), \(k\) is the ReLU smoothness/order parameter, and the hidden constant is independent of \(\underline h\). Since quasi-uniform sets with \(n\) points typically satisfy \(\underline h\asymp n^{-1/d}\), the bound becomes \(\|v_n\|_{H^r}\lesssim n^{(r-s)/d}\|v_n\|_{H^s}\). In a neural PDE solver, interpret \(s\) as the derivative order reliably measured by the collocation mesh and \(r\) as the derivative order appearing in a loss or stability estimate. The growth factor flags when high-order derivative losses will amplify coefficient noise and conditioning.

Implementation notes

1. Integration point: modify the derivative-residual loss and collocation scheduler in a PINN, neural operator, or coordinate MLP. The relevant quantities are feature-parameter separation \(\underline h\), derivative order \(r\) used by the PDE residual, and a lower-order diagnostic norm of order \(s\). 2. Pseudocode: initialize feature parameters \(\Theta\); compute \(\underline h=\min_{i\ne j}\|\theta_i-\theta_j\|\) after identifying antipodal duplicates; choose the largest candidate derivative order \(r<k+1/2\) satisfying \(\underline h^{-(r-s)}\le C_{\max}\). At each training phase, estimate lower-order residual norm \(E_s\) on a validation point set, compute the bound proxy \(B_r=\underline h^{-(r-s)}E_s\), and either use derivative loss \(L_r\) only when \(B_r\) is below a threshold or refine by adding well-separated features and collocation points. A practical loss is \(L=L_s+\lambda_rL_r\) with \(\lambda_r=\min(1,\tau/B_r)\). 3. The theorem supplies the exponent \(r-s\) and admissible range \(r<k+1/2\); the constant, actual Sobolev norms, and domain transfer must be estimated empirically. For nonspherical inputs, replace \(\underline h\) by minimum hidden-weight separation or a measured feature-kernel length scale. 4. First experiment: train width-128 and width-256 coordinate ReLU MLPs on a 2D Poisson or biharmonic problem with first-, second-, and fourth-order residual penalties. Compare fixed derivative losses against the adaptive rule at equal FLOPs. Track gradient-norm variance, residual-Jacobian condition number, dense-grid PDE residual, and final solution error. Success is that the adaptive rule prevents exploding derivative gradients and achieves equal or lower dense-grid error without increasing the collocation budget. A falsifying result is no reduction in instability or no correlation between predicted factor \(\underline h^{-(r-s)}\) and measured derivative amplification.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.