Unverified 2026

Extrapolation-Ratio Regularizer

Implementation & benchmark of arXiv:2607.10501 — Optimal Extrapolation Bounds for Sparse Fourier Sums

Usefulness6/10

Difficulty3/10

Novelty8/10

Regularization Stability Generalization Accuracy Loss Regularization Embedding Mlp Harmonic analysis Approximation theory Functional analysis Numerical analysis

Source paper: Optimal Extrapolation Bounds for Sparse Fourier Sums arXiv:2607.10501 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a differentiable regularizer to neural networks that learn sparse Fourier coefficients or trainable Fourier-feature frequencies. It penalizes predicted energy just outside the training interval when that energy exceeds the theorem-shaped envelope relative to observed in-domain L2 energy, discouraging cancellation patterns that fit the observed interval but explode nearby.

Formulas

$$\|g\|_{L^{2}[-1,1]}=\left(\int_{-1}^{1}|g(t)|^{2}\,dt\right)^{1/2},\qquad g(t)=\sum_{j=1}^{k}v_j e^{i\lambda_j t}$$

$$|g(1+\delta)|\leq O(k)\exp\!\bigl(O(k\sqrt{\delta})\bigr)\|g\|_{L^{2}[-1,1]},\qquad 0\leq\delta\leq1$$

$$R_{\delta}=\frac{\left(M^{-1}\sum_{m=1}^{M}|g(1+\delta_m)|^{2}\right)^{1/2}}{\left(N^{-1}\sum_{n=1}^{N}|g(t_n)|^{2}\right)^{1/2}+\varepsilon}$$

$$\mathcal{L}_{\mathrm{ext}}=\frac{1}{M}\sum_{m=1}^{M}\operatorname{softplus}\!\left(\log(R_{\delta_m}+\varepsilon)-\log\!\left(C_0k^p e^{C_1k\sqrt{\delta_m}}\right)\right)$$

Mathematical statement

The paper bounds the evaluation functional outside [-1,1] on the k-dimensional sparse Fourier model. For g(t)=sum_{j=1}^k v_j exp(i lambda_j t), the endpoint inequality gives an amplification ceiling C_0 k^p exp(C_1 k sqrt(delta)) relative to ||g||_{L^2[-1,1]}, where C_0,C_1,p are conservative versions of the paper's hidden constants. Define the empirical ratio R_delta=(M^{-1} sum_m |g(1+delta_m)|^2)^{1/2}/((N^{-1} sum_n |g(t_n)|^2)^{1/2}+epsilon), with t_n in [-1,1] and delta_m in [0,1]. Penalize only violations using softplus: L_ext=M^{-1} sum_m softplus(log(R_delta_m+epsilon)-log(C_0 k^p exp(C_1 k sqrt(delta_m)))). The regularizer requires no frequency-separation assumption.

Implementation notes

(1) Integration point: apply this loss during training to a sparse Fourier head, Fourier-feature encoder with trainable frequencies, spectral MLP, or neural field whose scalar output is evaluated on normalized coordinates [-1,1]. Compute it on a small probe batch rather than every full-resolution training sample. (2) Pseudocode: draw N in-domain points t_n uniformly from [-1,1]; draw M offsets delta_m from {0.01,0.05,0.1,0.25,0.5}; evaluate the current Fourier representation on t_n and 1+delta_m; compute E_in=sqrt(mean(abs(g_in)^2)+eps); compute R_m=sqrt(mean(abs(g_out_m)^2)+eps)/(E_in+eps); compute B_m=C0*k^p*exp(C1*k*sqrt(delta_m)); add lambda_ext*mean(softplus(log(R_m+eps)-log(B_m))) to the task loss. Sample both left and right boundaries for symmetric data. (3) The paper fixes the functional dependence but the extracted statement hides numerical constants. Estimate C0,C1,p in a no-gradient calibration sweep over random coefficients and frequencies, including clustered frequencies, using the 95th percentile of observed log ratios plus a safety margin. During training, log the true held-out ratios separately from the penalty. (4) First experiment: train a Fourier MLP on a 1D signal and a 2D image while withholding the outer 20% coordinate band. Compare lambda_ext in {0,1e-4,1e-3,1e-2} with an unregularized baseline at equal model size and training FLOPs. Report in-domain MSE, extrapolation MSE, maximum absolute output, gradient norm, and frequency sparsity. The desired result is lower outside-band MSE and peak overshoot at nearly unchanged in-domain loss, without increased optimization instability.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.