✗ Failed on benchmark 2026

Spectral quadrature features

Implementation & benchmark of arXiv:2609.00553 — Spectral Bounds for Kernel Quadrature

Usefulness7/10

Difficulty4/10

Novelty5/10

Memory Memory Accuracy Speedup Embedding Inference speedup Training Numerical analysis Spectral theory Linear algebra Approximation theory

Source paper: Spectral Bounds for Kernel Quadrature arXiv:2609.00553 ⓘ · analyzed Sep 2, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace random Fourier or random NTK features by a weighted deterministic quadrature rule for the kernel's feature integral. The resulting feature map uses the same linear-model interface as random features but can approximate the leading Gram-matrix eigenvalues substantially better at the same feature count, improving conditioning and reducing the width required for a target kernel approximation.

Formulas

$$K(x,y)=\int_{\mathbb{W}}G(w,x)\,\overline{G(w,y)}\,d\mu^{*}(w),\qquad x,y\in\mathbb{X}.$$

$$\widehat K_N(x,y)=\sum_{i=1}^{N}a_iG(w_i,x)\,\overline{G(w_i,y)},\qquad a_i\geq 0,\quad \sum_{i=1}^{N}a_i=1.$$

$$Z_{mi}=\sqrt{a_i}\,G(w_i,x_m),\qquad \widehat{\mathbf K}=ZZ^{*},\qquad \widehat K_{N}(x_m,x_n)=\sum_{i=1}^{N}Z_{mi}\overline{Z_{ni}}.$$

$$|\lambda_j(\mathbf K)-\lambda_j(\widehat{\mathbf K})|\leq\|\mathbf K-\widehat{\mathbf K}\|_2.$$

Mathematical statement

The paper's central construction is the integral kernel representation $K(x,y)=\int_{\mathbb{W}}G(w,x)\,\overline{G(w,y)}\,d\mu^{*}(w)$, where $\mathbb{W}$ is the feature-parameter domain, $\mu^{*}$ is a probability measure, and $G(w,x)$ is a possibly complex-valued feature function. A quadrature rule consists of nodes $w_i\in\mathbb{W}$ and nonnegative weights $a_i$ satisfying $\sum_{i=1}^{N}a_i=1$, producing $\widehat K_N(x,y)=\sum_{i=1}^{N}a_iG(w_i,x)\overline{G(w_i,y)}$. For data points $x_1,\ldots,x_M$, define $Z_{mi}=\sqrt{a_i}G(w_i,x_m)$; then the approximate Gram matrix is $\widehat{\mathbf K}=ZZ^{*}$, while $\mathbf K_{mn}=K(x_m,x_n)$. The spectral effect can be monitored with Weyl's inequality $|\lambda_j(\mathbf K)-\lambda_j(\widehat{\mathbf K})|\leq\|\mathbf K-\widehat{\mathbf K}\|_2$, where $\lambda_j$ is the $j$th ordered eigenvalue. The engineering objective is to choose quadrature nodes so that the leading eigenvalue errors are smaller than those obtained from iid Monte Carlo nodes, rather than merely minimizing pointwise kernel error.

Implementation notes

Integrate this at the input embedding or kernel layer of a kernelized model, a random-feature MLP, or a neural tangent-kernel approximation. Given inputs $X\in\mathbb R^{M\times d}$, first select a fixed quadrature rule $(w_i,a_i)_{i=1}^N$ for the feature domain: use Gauss-Hermite nodes for Gaussian Fourier features in low-dimensional Euclidean inputs, or spherical-design/quasi-uniform sphere nodes for features whose parameter domain is a sphere. Normalize weights to be nonnegative and sum to one. Construct the feature matrix with the following pseudocode: nodes, weights = quadrature_rule(N); weights = weights / weights.sum(); for minibatch Xb: F[:,i] = sqrt(weights[i]) * G(nodes[i], Xb); H = F @ theta, where $G$ is the original Fourier, sigmoid, ReLU, or other feature function and $\theta$ is the trainable linear readout. For complex Fourier features, either retain complex tensors or concatenate real and imaginary parts. The formula for $Z$ and $\widehat K=ZZ^*$ is computed exactly; quadrature nodes and weights are fixed before training. On a held-out subset, compute the exact kernel Gram matrix using the original integral formula or a very large reference feature set, then estimate $\lambda_j(\mathbf K)$ and $\lambda_j(\widehat{\mathbf K})$ with Lanczos. Also measure the operator-norm proxy, leading-eigenvalue relative error, effective rank, kernel ridge validation error, and condition number. The first cheap experiment should use MNIST or CIFAR-10 reduced to 20--50 PCA dimensions, compare $N\in\{64,128,256,512\}$ deterministic Gauss or spherical quadrature features against equal-width iid random Fourier features, and train the same linear ridge classifier. Success means lower top-20 eigenvalue error and equal or better test accuracy at the same $N$; a stronger result is achieving the random-feature accuracy with at least 2x fewer features, reducing both feature memory and matrix-multiplication cost.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 8/10

Built deterministic tensor Gauss-Hermite quadrature Fourier features for the Gaussian RBF kernel and compared them with iid Gaussian random Fourier features at matched widths. The claimed spectral phenomenon was observed: at 64 features, mean top-20 relative eigenvalue error was 0.125 for quadrature versus 0.231 for random features, and operator error was 2.23 versus 8.24; the trend was also favorable at 9, 36, and 64 features. The algebraic ZZ* identity, normalized nonnegative weights, and Weyl inequality check all passed, but the small kernel-ridge learning check favored random features (MSE 0.00064 versus 0.0357), so this is not an overall modeling win.

Agent confidence: 8/10

Baseline: iid Gaussian RFF: top-20 relative eigenvalue error 0.764, 0.624, 0.408, 0.231 at widths 9, 16, 36, 64; operator error 24.32, 17.50, 11.57, 8.24; 64-feature ridge MSE 0.00064 +/- 0.00082 over 20 seeds.
Idea: Gauss-Hermite quadrature: top-20 relative eigenvalue error 0.665, 0.525, 0.283, 0.125 at widths 9, 16, 36, 64; operator error 21.34, 9.90, 4.82, 2.23; 64-feature ridge MSE 0.0357.

Limitations: Only a 2D Gaussian RBF kernel, 120 synthetic points, tensor Gauss-Hermite rules, and a scalar small-data ridge task were tested. No PCA MNIST/CIFAR benchmark, higher-dimensional quadrature, classification, runtime/memory profiling, or learned/adaptive quadrature was evaluated; the ridge result is based on one fixed train/test split.

How to run: python3 spectral_quadrature_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2609.00553) Sep 2, 2026

No significant difference vs. the tuned baseline (+4.0%, p=0.4375, wins 4 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: fourier_mlp_shared
Paired seeds: 8
Baseline mean: 23.2698
Idea mean: 24.2013
Effect (Δ): 0.9314 (+4.0%; negative = idea better)
Wins: 4 / 8 paired seeds
p-value: 0.4375 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±10.5%
Mechanism: Not confirmed ✗
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

Only the built-in tabular Friedman#1 track was tested; vision, sequence, dynamics, and custom tracks were not evaluated. The quadrature rule used a fixed 10-dimensional product-style radial/directional construction rather than a high-quality dimension-specific spherical design, and no feature-count or runtime sweep was performed.

How to run: python3 spectral_quadrature_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 9/10

Implemented deterministic Gaussian spectral quadrature Fourier features and compared them against iid Gaussian Fourier features in the matched tabular Friedman#1 benchmark using the same downstream MLP, optimizer, epochs, batch size, and learning-rate union. Baseline test MSE was 23.270 versus 24.201 for the idea, giving idea-minus-baseline delta +0.9314 with permutation p=0.43755; the trained-model top-10 Gram eigenvalue error was also worse for quadrature (4.321 versus 1.423), so the mechanism prediction was not confirmed.

Baseline: Friedman#1 tabular, Fourier MLP, Adam 12 epochs, lr sweep {0.001, 0.003, 0.01}; best lr=0.003; 8-seed test MSE mean 23.26982045173645, std 2.8220737062934167.
Idea: Deterministic 64-feature radial generalized-Laguerre plus antipodal directional quadrature Fourier embedding, same MLP and sweep; best lr=0.003; 8-seed test MSE mean 24.201251745224, std 1.9567082483775788; delta +0.9314312934875488, p=0.43755. Trained-model spectral signature: baseline top-10 relative eigenvalue error 1.4234343767166138, idea 4.320810317993164, delta +2.8973759412765503, confirmed=false.

Benchmark result