# Sobolev-Orthogonal MLP Features

- ID: 1992
- Canonical URL: https://synthcore.org/idea/1992/sobolev-orthogonal-mlp-features
- API JSON: https://synthcore.org/api/idea/1992.json
- API Markdown: https://synthcore.org/api/idea/1992.md
- Verification status: unverified
- Source: [arXiv:2608.08397](https://arxiv.org/abs/2608.08397)
- Category: architecture
- Solves: stability, accuracy, generalization
- ML areas: mlp, embedding, regularization, initialization
- Math tags: approximation-theory, functional-analysis, differential-geometry, numerical-analysis
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Replace raw polynomial or Fourier-like features in a small MLP with basis functions orthonormal under a Sobolev inner product that jointly measures feature magnitude and input derivative magnitude. This explicitly controls feature smoothness while preserving decorrelation, potentially improving conditioning and reducing the need for large derivative-regularization coefficients.

## Mathematical statement

The paper studies monic polynomials orthogonal under the Sobolev inner product \(\langle P,Q\rangle_S=\int_{\mathbb R}P(x)Q(x)e^{-V(x)}\,dx+\lambda\int_{\mathbb R}P'(x)Q'(x)e^{-V(x)}\,dx\). Here \(P,Q\) are polynomials, \(P'\) and \(Q'\) are derivatives with respect to the scalar input \(x\), \(V\) is an even-degree polynomial with positive leading coefficient, \(e^{-V(x)}\) is the confining weight, and \(\lambda>0\) controls the derivative metric. The desired basis \(\{\Pi_n\}\) satisfies \(\langle \Pi_i,\Pi_j\rangle_S=\delta_{ij}\), so value covariance and derivative covariance are jointly controlled. Proposition 2.7 gives the auxiliary entire function \(F_k\) through the second-order ODE \((-1)^kV'(z)F_k'(z)-F_k''(z)+\lambda^{-1}F_k(z)=0\), where \(z\) is complex, \(k\) indexes a sector, and \(F_k(z)=1+\mathcal O(|z|^{2-D})\) in the sector as \(|z|\to\infty\), with \(D=\deg V\). For an MVP, the orthogonal basis can be computed directly from the Sobolev Gram matrix; the ODE is an optional stable generator for higher degrees.

## Key formulas

- $$\langle P,Q\rangle_S=\int_{\mathbb R}P(x)Q(x)e^{-V(x)}\,dx+\lambda\int_{\mathbb R}P'(x)Q'(x)e^{-V(x)}\,dx$$
- $$\langle \Pi_i,\Pi_j\rangle_S=\delta_{ij}$$
- $$(-1)^kV'(z)F_k'(z)-F_k''(z)+\lambda^{-1}F_k(z)=0,\qquad F_k(z)=1+\mathcal O(|z|^{2-D})$$
- $$G_{ij}=\int_{\mathbb R}x^{i+j}e^{-V(x)}\,dx+\lambda ij\int_{\mathbb R}x^{i+j-2}e^{-V(x)}\,dx,\qquad \phi(x)=L^{-1}[1,x,\ldots,x^{N-1}]^{\mathsf T},\quad LL^{\mathsf T}=G$$

## Implementation notes

Integrate this at the input of a scalar or low-dimensional MLP, before the first learned linear layer. For each scalar normalized coordinate x in [-1,1], choose an even confining potential such as V(x)=a x^4+b x^2 with a>0, and degree N-1. Construct the Sobolev Gram matrix G using numerical Gauss-Hermite or Gauss-Legendre quadrature: for indices i,j starting at zero, compute G_ij=E[x^(i+j)]+lambda*i*j*E[x^(i+j-2)], with the derivative term set to zero when i=0 or j=0. Add a small epsilon I, perform Cholesky factorization G=LL^T, and define phi(x)=L^{-1}[1,x,...,x^(N-1)]^T. For vector inputs, concatenate one-dimensional bases per coordinate or use a tensor-product basis only for very small N. Feed phi(x) to a standard MLP and initialize its first linear layer with fan-in scaling. During training, optionally learn lambda through softplus(lambda_raw), but recompute the basis only every K steps or keep lambda fixed in the first experiment. The mathematical computation is the Sobolev Gram matrix and Cholesky orthogonalization; empirical choices are V, degree N, lambda, and quadrature resolution. As a cheap test, compare a 2-layer MLP using raw monomials, ordinary weighted orthogonal polynomials, random Fourier features, and Sobolev features on noisy 1D and 2D regression, such as sinusoid-plus-high-frequency perturbations. Measure the condition number of the first-layer feature covariance, gradient variance, validation MSE, and sensitivity to learning rate. Success means substantially lower feature and gradient condition numbers and lower validation error or faster loss descent at equal parameter count, especially when the target is smooth and training inputs are unevenly distributed.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.