Nested Bandwidth Feature Curriculum
Implementation & benchmark of arXiv:2609.03401 — Spectral Convergence of Random Feature Method in Multiple Dimensions
Source paper: Spectral Convergence of Random Feature Method in Multiple Dimensions arXiv:2609.03401 ⓘ · analyzed Sep 4, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
For bandlimited or progressively higher-frequency targets, construct a nested random Fourier layer whose frequency window grows according to the paper's bandwidth laws instead of sampling all frequencies from one fixed range. Train with a low-bandwidth subset first, then activate additional frequency blocks. This creates a mathematically motivated spectral curriculum and may reduce early optimization difficulty while preserving high-frequency accuracy.
Formulas
Mathematical statement
Let U be an L2 function on R^d whose Fourier transform is supported in the cube Q_S, let the spatial domain Omega lie inside a translated cube Q_R, and let J >= max{2 S R, 2} be an integer approximation order. The paper defines the leverage quantity lambda_(J,S) = C_(a,d) (pi/S)^(d a) J^(-1) (1+S R)^a (S R/J)^(2 a J), where a is in (0,1), C_(a,d) >= 1 depends on a and d, S is the target bandwidth, and R is the spatial radius. It requires N >= 6 J^d log(28 J^d / delta), where N is the number of sampled features and delta is the failure probability. For an ultra-analytic schedule, the paper gives S_J = (J log J)^(1/s) / (4 R_*), where s > 1 is a regularity parameter and R_* is a domain-dependent radius. For a bandlimited target, use the terminal bandwidth S directly. Adapt these results into nested windows Q_(S_1) subset Q_(S_2) subset ... .
Implementation notes
1. Integration point: modify the Fourier-feature embedding before a coordinate MLP or neural operator. Partition frequencies into nested blocks W_1 subset W_2 subset ... , where block k contains frequencies sampled uniformly from Q_(S_k) = [-S_k,S_k]^d. Keep all blocks in memory but gate later blocks off during early training. 2. Pseudocode: estimate the domain radius R, choose delta = 0.05, and select orders J_k = 2^k J_0. Set S_k = (J_k log J_k)^(1/s) / (4 R_*) for ultra-analytic data, or use a known terminal bandwidth S for bandlimited data. Allocate N_k features so cumulative count satisfies N_k >= 6 J_k^d log(28 J_k^d / delta). At epoch e, activate the smallest k allowed by the curriculum schedule, compute Z_k = sqrt(2) cos(X W_k + b_k), concatenate active blocks, and rescale each block by its running RMS. 3. Compute S_k, J_k, and the sample-size targets from the displayed formulas. Estimate unknown R_*, s, and terminal bandwidth from a validation Fourier spectrum. Use lambda_(J,S) as a conditioning diagnostic; when it becomes very small, apply ridge stabilization to the active feature head using G + rho I, with rho proportional to lambda_(J,S) after clipping to a numerically safe interval. 4. First cheap experiment: regress 2D synthetic functions with known Fourier bandwidth on a small MLP using 64 to 1024 total features. Compare the nested curriculum against all features active from step zero and fixed-window RFF at equal total FLOPs. Track loss during the first 20 percent of updates, final test error, gradient-norm spikes, and activation-block utilization. Success means faster early loss descent, fewer optimizer instabilities, and lower high-frequency test error at the same final feature count.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.