Sobolev-Orthogonal MLP Features
Implementation & benchmark of arXiv:2608.08397 — A Riemann-Hilbert representation for Sobolev orthogonal polynomials
Source paper: A Riemann-Hilbert representation for Sobolev orthogonal polynomials arXiv:2608.08397 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace raw polynomial or Fourier-like features in a small MLP with basis functions orthonormal under a Sobolev inner product that jointly measures feature magnitude and input derivative magnitude. This explicitly controls feature smoothness while preserving decorrelation, potentially improving conditioning and reducing the need for large derivative-regularization coefficients.
Formulas
Mathematical statement
The paper studies monic polynomials orthogonal under the Sobolev inner product \(\langle P,Q\rangle_S=\int_{\mathbb R}P(x)Q(x)e^{-V(x)}\,dx+\lambda\int_{\mathbb R}P'(x)Q'(x)e^{-V(x)}\,dx\). Here \(P,Q\) are polynomials, \(P'\) and \(Q'\) are derivatives with respect to the scalar input \(x\), \(V\) is an even-degree polynomial with positive leading coefficient, \(e^{-V(x)}\) is the confining weight, and \(\lambda>0\) controls the derivative metric. The desired basis \(\{\Pi_n\}\) satisfies \(\langle \Pi_i,\Pi_j\rangle_S=\delta_{ij}\), so value covariance and derivative covariance are jointly controlled. Proposition 2.7 gives the auxiliary entire function \(F_k\) through the second-order ODE \((-1)^kV'(z)F_k'(z)-F_k''(z)+\lambda^{-1}F_k(z)=0\), where \(z\) is complex, \(k\) indexes a sector, and \(F_k(z)=1+\mathcal O(|z|^{2-D})\) in the sector as \(|z|\to\infty\), with \(D=\deg V\). For an MVP, the orthogonal basis can be computed directly from the Sobolev Gram matrix; the ODE is an optional stable generator for higher degrees.
Implementation notes
Integrate this at the input of a scalar or low-dimensional MLP, before the first learned linear layer. For each scalar normalized coordinate x in [-1,1], choose an even confining potential such as V(x)=a x^4+b x^2 with a>0, and degree N-1. Construct the Sobolev Gram matrix G using numerical Gauss-Hermite or Gauss-Legendre quadrature: for indices i,j starting at zero, compute G_ij=E[x^(i+j)]+lambda*i*j*E[x^(i+j-2)], with the derivative term set to zero when i=0 or j=0. Add a small epsilon I, perform Cholesky factorization G=LL^T, and define phi(x)=L^{-1}[1,x,...,x^(N-1)]^T. For vector inputs, concatenate one-dimensional bases per coordinate or use a tensor-product basis only for very small N. Feed phi(x) to a standard MLP and initialize its first linear layer with fan-in scaling. During training, optionally learn lambda through softplus(lambda_raw), but recompute the basis only every K steps or keep lambda fixed in the first experiment. The mathematical computation is the Sobolev Gram matrix and Cholesky orthogonalization; empirical choices are V, degree N, lambda, and quadrature resolution. As a cheap test, compare a 2-layer MLP using raw monomials, ordinary weighted orthogonal polynomials, random Fourier features, and Sobolev features on noisy 1D and 2D regression, such as sinusoid-plus-high-frequency perturbations. Measure the condition number of the first-layer feature covariance, gradient variance, validation MSE, and sensitivity to learning rate. Success means substantially lower feature and gradient condition numbers and lower validation error or faster loss descent at equal parameter count, especially when the target is smooth and training inputs are unevenly distributed.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.