△ Mechanism confirmed, baseline not beaten 2026

Regularity-Matched Random Fourier Layer

Implementation & benchmark of arXiv:2609.03401 — Spectral Convergence of Random Feature Method in Multiple Dimensions

Usefulness7/10

Difficulty4/10

Novelty5/10

Architecture Accuracy Sample efficiency Stability Embedding Mlp Initialization Neural operator Approximation theory Spectral theory Probability Linear algebra

Source paper: Spectral Convergence of Random Feature Method in Multiple Dimensions arXiv:2609.03401 ⓘ · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the usual isotropic Gaussian random Fourier features with a frequency distribution matched to the expected spectral regularity of the target function. For coordinate fields, operator-learning maps, or PDE solution surrogates, this should place more features where the target Fourier energy lies and improve approximation at the same feature count. Stabilize the resulting feature matrix with whitening or ridge regression because spectral accuracy can create severe ill-conditioning.

Formulas

$$L_N := \frac{N}{\log(N/\delta)}$$

$$\mathrm{d}\tau_{\mathrm{Sob}}(w) \propto (1+|w|^{2})^{-\bar{s}}\mathrm{d}w, \qquad \|f-f_N\| \text{ rate} \sim L_N^{-(s-t)/d}$$

$$\mathrm{d}\tau_{\mathrm{Gevrey}}(w) \propto \exp(-2\bar{\kappa}|w|^{1/s})\mathrm{d}w, \qquad \|f-f_N\| \text{ rate} \sim \exp(-cL_N^{1/(sd)})$$

$$\mathrm{d}\tau_{\mathrm{ultra}}(w) \propto \exp(-2\bar{\kappa}|w|^{s})\mathrm{d}w, \qquad \|f-f_N\| \text{ rate} \sim \exp(-cL_N^{1/d}\log L_N)$$

Mathematical statement

Let N be the number of sampled features, delta in (0,1) the failure probability, d the input dimension, and L_N := N / log(N/delta). The paper's rate table states that Sobolev targets of smoothness s, measured in an error norm indexed by t, use d tau_Sob(w) proportional to (1+|w|^2)^(-bar{s}) d w and achieve rate L_N^(-(s-t)/d). Gevrey targets with parameter s >= 1 use d tau_Gevrey(w) proportional to exp(-2 bar{kappa}|w|^(1/s)) d w and achieve exp(-c L_N^(1/(s d))). Ultra-analytic targets with s > 1 use d tau_ultra(w) proportional to exp(-2 bar{kappa}|w|^s) d w and achieve exp(-c L_N^(1/d) log L_N). Here bar{s}, bar{kappa}, and c are positive constants determined by the distribution and target class. Implement phi_w(x) = sqrt(2) cos(w^T x + b), with b uniform on [0,2 pi], and train a head on the resulting feature matrix. The extracted text states that the same spectral mechanism causing high accuracy also causes severe condition-number growth, so empirical whitening is part of the adaptation.

Implementation notes

1. Integration point: use this as the frozen input projection in a coordinate MLP, neural operator, PINN, or implicit neural representation. Given inputs X in R^(B by d), sample N frequencies once at initialization, compute Z_bn = sqrt(2) cos(X_b dot w_n + b_n), and feed Z to a trainable linear or MLP head. 2. Pseudocode: choose a target regularity mode. For Sobolev, sample each w_n from the normalized density proportional to (1+||w||^2)^(-bar{s}); for Gevrey, sample from the density proportional to exp(-2 bar{kappa} ||w||^(1/s)); for ultra-analytic, sample from exp(-2 bar{kappa} ||w||^s). Draw b_n uniformly from [0, 2 pi], form Z, estimate the empirical Gram matrix G = Z^T Z / B + epsilon I, and replace Z by Z G^(-1/2) using an eigendecomposition or QR factorization. Train the downstream head first and optionally unfreeze frequencies later. 3. Compute the distributions and L_N exactly from the paper; estimate the Gram spectrum empirically and choose epsilon = 10^(-5) trace(G)/N. 4. First experiment: fit 2D Sobolev, Gevrey, and bandlimited synthetic functions on a 64 by 64 grid using a linear head with 256, 512, and 1024 features. Compare Gaussian RFF, uniform-window RFF, and matched distributions at equal feature count and FLOPs, with and without whitening. Measure test L2 error, convergence slope versus N, condition number, and wall-clock stability. Success means lower error at fixed N, a steeper log-error curve for smooth targets, and no instability after whitening.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built a fixed-seed Regularity-Matched RFF MVP with Gaussian, uniform-window, and Gevrey-matched frequency sampling, ridge linear fitting, empirical whitening, and distribution/conditioning sanity checks. The matched layer clearly improved low-feature approximation: RMSE was 0.0501 vs 0.402 for Gaussian at N=64, and 0.00234 vs 0.01355 at N=128 after whitening. The effect is not universal—Gaussian caught up or exceeded it at N>=256—and whitening reduced error but did not fully eliminate severe Gram ill-conditioning, so this is a promising low-N signal rather than a broad win.

Agent confidence: 8/10

Baseline: Gaussian RFF RMSE: N=64 0.4020, N=128 0.01355, N=256 0.00010, N=512 approximately 0; raw Gram condition numbers approximately 3.9e2, 3.6e6, 8.2e15, 1.2e16.
Idea: Gevrey-matched RFF RMSE: N=64 0.05006 raw / 0.04531 whitened, N=128 0.00954 raw / 0.00234 whitened, N=256 0.00353 raw / 0.00052 whitened, N=512 0.00034 raw / 0.00002 whitened; raw Gram condition numbers approximately 5e12, 2.5e16, 4.6e16, 9.5e16. Empirical radii had Gaussian median 3.53 versus matched median 1.12, consistent with concentrating frequencies near the smooth target’s spectral region.

Limitations: Only a single analytic 2D synthetic target was tested, with three random repetitions per configuration; no PDE, operator-learning, grid-based 64x64 experiment, FLOP-normalized timing, ultra-analytic distribution, or larger-dimensional target was evaluated. The current ridge-style whitening improves prediction but does not guarantee a well-conditioned transformed Gram matrix for nearly redundant features.

How to run: python3 experiment.py > run_output.txt

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.03401) Sep 4, 2026

Ahead of the baseline (−1.6%) but not statistically significant (p=0.4471, wins 5 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 26.4281
Idea mean: 25.9967
Effect (Δ): -0.4313 (−1.6%; negative = idea better)
Wins: 5 / 8 paired seeds
p-value: 0.4471 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±5.2%
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally appropriate registered tabular track was tested. Vision, sequence, dynamics, PDE/custom tracks, larger feature counts, alternative Sobolev or ultra-analytic distributions, and unwhitened variants were not tested.

How to run: python3 bench_rff.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented and ran the full stage-2 protocol on the registered built-in tabular track with paired Gaussian-RFF baseline and Gevrey-matched whitened RFF idea using the same MLP system. The idea improved mean test MSE by 0.4313, but the paired permutation p-value was 0.4471, so there was no significant win. The trained-model mechanism signature was quantitatively confirmed, although the effect did not yield a reliable task-metric improvement.

Baseline: Best Gaussian-RFF configuration lr=0.003; mean MSE 26.428078.
Delta mean: -0.4313347339630127
Idea: Best Gevrey-matched whitened RFF configuration lr=0.003, kappa=0.5; mean MSE 25.996743.
Idea wins: 5
N pairs: 8
P value: 0.4471

Benchmark result