# Variable-Projection Adaptive Fourier Layer

- ID: 2519
- Canonical URL: https://synthcore.org/idea/2519/variable-projection-adaptive-fourier-layer
- API JSON: https://synthcore.org/api/idea/2519.json
- API Markdown: https://synthcore.org/api/idea/2519.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.18380](https://arxiv.org/abs/2608.18380)
- Category: architecture
- Solves: accuracy, sample-efficiency, stability
- ML areas: embedding, loss, neural-operator, architecture
- Math tags: optimization, linear-algebra, approximation-theory, pde
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace fixed random Fourier features in a coordinate network or PINN with a small complex Fourier dictionary whose propagation directions are learned from a weighted residual. Given directions, solve the linear feature coefficients exactly or by ridge regression, and optimize only the directions in the outer loop. This should represent low-directional-complexity fields with fewer features and avoid wasting gradient updates on coefficients that can be fitted analytically.

## Mathematical statement

The paper's Part B construction minimizes a skeleton residual over both a Trefftz field and its directions, then eliminates the linear coefficient variables by variable projection. Let x_i in R^d be sampled coordinates, y_i in C^q be target values or residual targets, theta=(theta_1,...,theta_M) be nonlinear direction parameters, d_m(theta_m) in C^d be complex propagation directions, k be a frequency, and B_theta in C^{N x M} be the feature matrix with (B_theta)_{im}=exp(i k d_m(theta_m)^T x_i). Let W in R^{N x N} be a positive diagonal sample-weight matrix and lambda>=0 a ridge parameter. The inner linear problem is c*(theta)=argmin_c ||W^{1/2}(B_theta c-y)||_2^2+lambda||c||_2^2, with solution c*(theta)=(B_theta^*WB_theta+lambda I)^{-1}B_theta^*Wy. The reduced objective is Phi(theta)=||W^{1/2}(B_theta c*(theta)-y)||_2^2+lambda||c*(theta)||_2^2. For a Helmholtz feature, Delta exp(i k d^T x)=-k^2(d^T d)exp(i k d^T x); imposing d^T d=1 makes the feature satisfy the homogeneous Helmholtz equation. The paper's local quadratic-growth result motivates Phi(theta)>=Phi(theta*)+(mu/2)||theta-theta*||_2^2 in a fixed-rank identifiable neighborhood, where mu>0 is local curvature. Direction redundancy is detected through the weighted Gramian G=B_theta^*WB_theta.

## Key formulas

- $$c^*(\theta)=\arg\min_{c\in\mathbb{C}^M}\left\|W^{1/2}(B_\theta c-y)\right\|_2^2+\lambda\|c\|_2^2=(B_\theta^*WB_\theta+\lambda I)^{-1}B_\theta^*Wy.$$
- $$\Phi(\theta)=\left\|W^{1/2}(B_\theta c^*(\theta)-y)\right\|_2^2+\lambda\left\|c^*(\theta)\right\|_2^2.$$
- $$(B_\theta)_{im}=\exp\!\left(i k\,d_m(\theta_m)^\top x_i\right),\qquad \Delta\exp(i k d_m^\top x)=-k^2(d_m^\top d_m)\exp(i k d_m^\top x).$$
- $$\Phi(\theta)\geq \Phi(\theta^*)+\frac{\mu}{2}\|\theta-\theta^*\|_2^2\quad\text{for }\theta\text{ in a fixed-rank identifiable neighborhood of }\theta^*.$$

## Implementation notes

Integrate this as the first layer of a coordinate MLP, PINN, or neural operator that maps coordinates x to a field value. Maintain M complex directions d_m and use features z_m(x)=exp(i k d_m^T x); concatenate real and imaginary parts with ordinary MLP features. For the MVP, make the Fourier-feature output linear: c is solved analytically at every outer step, while theta is optimized by Adam or L-BFGS. At each iteration, sample coordinates x_i and targets or PDE residual equations y_i; construct B_theta with B[i,m]=exp(1j*k*dot(d_m,x_i)); form A=B.conj().T@W@B+lambda*I and b=B.conj().T@W@y; solve A c=b using Cholesky rather than an explicit inverse; compute Phi; backpropagate through the linear solve to theta; and update theta. Every K steps compute G=B.conj().T@W@B, eigendecompose G, and prune features whose associated subspace has eigenvalue below tau*lambda_max(G). Optionally birth a new direction at the largest residual Fourier correlation. Parameterize real propagating directions in two dimensions by d=(cos(theta),sin(theta)); for evanescent waves allow theta=a+ib and evaluate complex sine and cosine in double precision. The paper supplies the residual minimization, variable projection, quadratic-growth intuition, and Gramian cutoff; estimate W, lambda, tau, and local curvature empirically. First test on a 2D Helmholtz field made from M=4, 8, or 16 plane waves, comparing a fixed random Fourier-feature PINN and a SIREN at equal feature count. Measure relative L2 field error, boundary/PDE residual, number of retained features, and wall-clock time. Success is lower error with fewer features or at least 2x faster residual reduction at equal parameter count; failure is inability to recover planted directions or unstable Gramian pruning.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a NumPy/SciPy variable-projection Fourier MVP with complex ridge solves, direction optimization, Gramian inspection, and a fixed-random-feature comparison. Two quantitative mechanism checks passed: ridge objectives increased monotonically with lambda, and the near-duplicate Gramian eigenvalue scaled quadratically with angular separation (observed exponent 1.982 versus predicted 2.0). Direction recovery failed and the optimized model did not beat the fixed random baseline, so the full mechanism was not demonstrated.

### Mechanism check

- Verdict: Built a NumPy/SciPy variable-projection Fourier MVP with complex ridge solves, direction optimization, Gramian inspection, and a fixed-random-feature comparison. Two quantitative mechanism checks passed: ridge objectives increased monotonically with lambda, and the near-duplicate Gramian eigenvalue scaled quadratically with angular separation (observed exponent 1.982 versus predicted 2.0). Direction recovery failed and the optimized model did not beat the fixed random baseline, so the full mechanism was not demonstrated.
- Confidence: 9/10
- Limitations: Only a small 2D synthetic regression was tested; no PINN/PDE residual training, MLP integration, adaptive feature birth or pruning during optimization, weighted nonuniform samples, L-BFGS/Adam comparison, or GPU implementation was tested. The direction optimizer was derivative-free and multimodal, so failed recovery may reflect optimization difficulty as well as method limitations.

## Artifacts

- [report.md](https://synthcore.org/code/969/report.md)
- [results.json](https://synthcore.org/code/969/results.json)
- [vp_fourier_mvp.py](https://synthcore.org/code/969/vp_fourier_mvp.py)
- [Download all files as ZIP](https://synthcore.org/download/969)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.