# Nested Bandwidth Feature Curriculum

- ID: 3105
- Canonical URL: https://synthcore.org/idea/3105/nested-bandwidth-feature-curriculum
- API JSON: https://synthcore.org/api/idea/3105.json
- API Markdown: https://synthcore.org/api/idea/3105.md
- Verification status: unverified
- Source: [arXiv:2609.03401](https://arxiv.org/abs/2609.03401)
- Category: training
- Solves: accuracy, sample-efficiency, stability
- ML areas: embedding, mlp, curriculum, training, neural-operator
- Math tags: approximation-theory, spectral-theory, probability, linear-algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

For bandlimited or progressively higher-frequency targets, construct a nested random Fourier layer whose frequency window grows according to the paper's bandwidth laws instead of sampling all frequencies from one fixed range. Train with a low-bandwidth subset first, then activate additional frequency blocks. This creates a mathematically motivated spectral curriculum and may reduce early optimization difficulty while preserving high-frequency accuracy.

## Mathematical statement

Let U be an L2 function on R^d whose Fourier transform is supported in the cube Q_S, let the spatial domain Omega lie inside a translated cube Q_R, and let J >= max{2 S R, 2} be an integer approximation order. The paper defines the leverage quantity lambda_(J,S) = C_(a,d) (pi/S)^(d a) J^(-1) (1+S R)^a (S R/J)^(2 a J), where a is in (0,1), C_(a,d) >= 1 depends on a and d, S is the target bandwidth, and R is the spatial radius. It requires N >= 6 J^d log(28 J^d / delta), where N is the number of sampled features and delta is the failure probability. For an ultra-analytic schedule, the paper gives S_J = (J log J)^(1/s) / (4 R_*), where s > 1 is a regularity parameter and R_* is a domain-dependent radius. For a bandlimited target, use the terminal bandwidth S directly. Adapt these results into nested windows Q_(S_1) subset Q_(S_2) subset ... .

## Key formulas

- $$\lambda_{J,S}=C_{a,d}\left(\frac{\pi}{S}\right)^{da}J^{-1}(1+SR)^{a}\left(\frac{SR}{J}\right)^{2aJ}$$
- $$N\ge 6J^{d}\log\left(\frac{28J^{d}}{\delta}\right)$$
- $$S_J=\frac{(J\log J)^{1/s}}{4R_*}$$
- $$\widehat U(\xi)=0\quad\text{for }\xi\notin Q_S,\qquad Q_S=[-S,S]^d$$

## Implementation notes

1. Integration point: modify the Fourier-feature embedding before a coordinate MLP or neural operator. Partition frequencies into nested blocks W_1 subset W_2 subset ... , where block k contains frequencies sampled uniformly from Q_(S_k) = [-S_k,S_k]^d. Keep all blocks in memory but gate later blocks off during early training. 2. Pseudocode: estimate the domain radius R, choose delta = 0.05, and select orders J_k = 2^k J_0. Set S_k = (J_k log J_k)^(1/s) / (4 R_*) for ultra-analytic data, or use a known terminal bandwidth S for bandlimited data. Allocate N_k features so cumulative count satisfies N_k >= 6 J_k^d log(28 J_k^d / delta). At epoch e, activate the smallest k allowed by the curriculum schedule, compute Z_k = sqrt(2) cos(X W_k + b_k), concatenate active blocks, and rescale each block by its running RMS. 3. Compute S_k, J_k, and the sample-size targets from the displayed formulas. Estimate unknown R_*, s, and terminal bandwidth from a validation Fourier spectrum. Use lambda_(J,S) as a conditioning diagnostic; when it becomes very small, apply ridge stabilization to the active feature head using G + rho I, with rho proportional to lambda_(J,S) after clipping to a numerically safe interval. 4. First cheap experiment: regress 2D synthetic functions with known Fourier bandwidth on a small MLP using 64 to 1024 total features. Compare the nested curriculum against all features active from step zero and fixed-window RFF at equal total FLOPs. Track loss during the first 20 percent of updates, final test error, gradient-norm spikes, and activation-block utilization. Success means faster early loss descent, fewer optimizer instabilities, and lower high-frequency test error at the same final feature count.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.