Unverified 2026

Nested Bandwidth Feature Curriculum

Implementation & benchmark of arXiv:2609.03401 — Spectral Convergence of Random Feature Method in Multiple Dimensions

Usefulness6/10
Difficulty5/10
Novelty6/10

Source paper: Spectral Convergence of Random Feature Method in Multiple Dimensions arXiv:2609.03401 · analyzed Sep 4, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

For bandlimited or progressively higher-frequency targets, construct a nested random Fourier layer whose frequency window grows according to the paper's bandwidth laws instead of sampling all frequencies from one fixed range. Train with a low-bandwidth subset first, then activate additional frequency blocks. This creates a mathematically motivated spectral curriculum and may reduce early optimization difficulty while preserving high-frequency accuracy.

Formulas

$$\lambda_{J,S}=C_{a,d}\left(\frac{\pi}{S}\right)^{da}J^{-1}(1+SR)^{a}\left(\frac{SR}{J}\right)^{2aJ}$$
$$N\ge 6J^{d}\log\left(\frac{28J^{d}}{\delta}\right)$$
$$S_J=\frac{(J\log J)^{1/s}}{4R_*}$$
$$\widehat U(\xi)=0\quad\text{for }\xi\notin Q_S,\qquad Q_S=[-S,S]^d$$

Mathematical statement

Let U be an L2 function on R^d whose Fourier transform is supported in the cube Q_S, let the spatial domain Omega lie inside a translated cube Q_R, and let J >= max{2 S R, 2} be an integer approximation order. The paper defines the leverage quantity lambda_(J,S) = C_(a,d) (pi/S)^(d a) J^(-1) (1+S R)^a (S R/J)^(2 a J), where a is in (0,1), C_(a,d) >= 1 depends on a and d, S is the target bandwidth, and R is the spatial radius. It requires N >= 6 J^d log(28 J^d / delta), where N is the number of sampled features and delta is the failure probability. For an ultra-analytic schedule, the paper gives S_J = (J log J)^(1/s) / (4 R_*), where s > 1 is a regularity parameter and R_* is a domain-dependent radius. For a bandlimited target, use the terminal bandwidth S directly. Adapt these results into nested windows Q_(S_1) subset Q_(S_2) subset ... .

Implementation notes

1. Integration point: modify the Fourier-feature embedding before a coordinate MLP or neural operator. Partition frequencies into nested blocks W_1 subset W_2 subset ... , where block k contains frequencies sampled uniformly from Q_(S_k) = [-S_k,S_k]^d. Keep all blocks in memory but gate later blocks off during early training. 2. Pseudocode: estimate the domain radius R, choose delta = 0.05, and select orders J_k = 2^k J_0. Set S_k = (J_k log J_k)^(1/s) / (4 R_*) for ultra-analytic data, or use a known terminal bandwidth S for bandlimited data. Allocate N_k features so cumulative count satisfies N_k >= 6 J_k^d log(28 J_k^d / delta). At epoch e, activate the smallest k allowed by the curriculum schedule, compute Z_k = sqrt(2) cos(X W_k + b_k), concatenate active blocks, and rescale each block by its running RMS. 3. Compute S_k, J_k, and the sample-size targets from the displayed formulas. Estimate unknown R_*, s, and terminal bandwidth from a validation Fourier spectrum. Use lambda_(J,S) as a conditioning diagnostic; when it becomes very small, apply ridge stabilization to the active feature head using G + rho I, with rho proportional to lambda_(J,S) after clipping to a numerically safe interval. 4. First cheap experiment: regress 2D synthetic functions with known Fourier bandwidth on a small MLP using 64 to 1024 total features. Compare the nested curriculum against all features active from step zero and fixed-window RFF at equal total FLOPs. Track loss during the first 20 percent of updates, final test error, gradient-norm spikes, and activation-block utilization. Success means faster early loss descent, fewer optimizer instabilities, and lower high-frequency test error at the same final feature count.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.