Sobolev-Spectral Degree Curriculum
Implementation & benchmark of arXiv:2607.11921 — Near-Optimal Learning of Gaussian Sobolev Operators
Source paper: Near-Optimal Learning of Gaussian Sobolev Operators arXiv:2607.11921 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Train polynomial interaction features in increasing Hermite degree and activate a new degree only when the previous spectral shell is fitted. This turns the paper's spectral approximation behavior into a curriculum and explicit regularizer, preventing high-order interaction parameters from amplifying noise before the low-order Gaussian structure is learned.
Formulas
Mathematical statement
Let S_p={alpha in N_0^{d_X}: ||alpha||_1<=p} be the multivariate Hermite index set of total degree at most p, and let Delta_p=S_p\setminus S_{p-1} be the degree-p shell. Proposition 5.1 studies f_b(x)=|<a,x>|^b under a Gaussian measure varrho and states that f_b belongs to the Gaussian Sobolev space H^k_varrho but not H^{k+1}_varrho exactly when k-1/2<b<=k+1/2. This identifies finite smoothness as the source of slower Hermite coefficient decay. Use a shell-weighted loss with coefficient penalty increasing by degree, or add shells adaptively based on validation residuals. The coefficients C_Delta_q are the columns of the latent-output coefficient matrix belonging to shell Delta_q, r is a tunable smoothness-penalty exponent, rho is ridge strength, and epsilon prevents division by zero. The paper supplies the mathematical motivation that higher Sobolev order k yields faster Hermite approximation convergence; r and k should be estimated rather than treated as known theorem constants.
Implementation notes
(1) Integration point: apply this to the Hermite head's coefficient matrix, or to a polynomial-feature adapter inserted between a transformer or MLP representation and a low-dimensional prediction head. Standardize and optionally whiten the input representation so its coordinates are approximately Gaussian before evaluating Hermite features. (2) Pseudocode: initialize p=0 and C on S_0; at each stage construct z, phi_Sp, and shell features phi_Deltap; optimize L_p for E epochs; compute validation error and R_p=||C_Deltap||_F^2/(||C_Sp||_F^2+epsilon); if R_p>tau and validation error improves, append the next shell, initialize its coefficients to zero, and continue; otherwise stop or increase rho. Use minibatch gradient descent or closed-form ridge solves for small latent dimensions. The probabilists' Hermite recursion is He_0=1, He_1=u, He_{n+1}(u)=u He_n(u)-n He_{n-1}(u). (3) Estimate r or k from the validation slope of shell energies versus degree; calibrate tau by fitting the same model to shuffled targets. (4) First experiment: use a 10-dimensional Gaussian-input regression problem with targets |a^T x|^b for b=1.2, 2.2, and 4.2, with 100, 500, and 2,000 samples. Compare adaptive degree selection against fixed-degree Hermite heads and a two-layer MLP. The expected signal is less high-degree overfitting, lower test MSE at small sample sizes, and selected degree increasing with b. On a neural-operator benchmark, success is lower validation error and smoother training at identical feature count.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.