# Rank-Safe Variable-Projection Gauss-Newton

- ID: 521
- Canonical URL: https://synthcore.org/idea/521/rank-safe-variable-projection-gauss-newton
- API JSON: https://synthcore.org/api/idea/521.json
- API Markdown: https://synthcore.org/api/idea/521.md
- Verification status: beats_baseline
- Source: [arXiv:2607.07506](https://arxiv.org/abs/2607.07506)
- Category: optimization
- Solves: speedup, stability, accuracy
- ML areas: optimizer, mlp, training-dynamics, loss
- Math tags: optimization, linear-algebra, numerical-analysis
- Ratings: usefulness 6/10; difficulty 6/10; novelty 5/10

## Idea description

Separate a neural network into nonlinear hidden parameters and a linear output layer. Solve the output layer exactly by least squares, then update hidden parameters with a truncated-pseudoinverse Gauss-Newton step that discards numerically singular directions.

## Mathematical statement

Let \(\phi_\theta(x)\in\mathbb{R}^{m}\) be the feature vector produced by hidden parameters \(\theta\), let \(w\in\mathbb{R}^{m\times q}\) be the linear output matrix, and let \(A\) denote the sampled observation, differential, or residual operator. With stacked targets \(y\), define \(\Phi_\theta\) as the feature matrix and \(M_\theta=A\Phi_\theta\). The least-squares objective is \(L(w,\theta)=\frac{1}{2}\|M_\theta w-y\|_2^2\). For fixed \(\theta\), compute the variable-projection solution using the singular value decomposition \(M_\theta=U\Sigma V^T\): \(w^*(\theta)=V\Sigma_\tau^+U^Ty\), where \((\Sigma_\tau^+)_{ii}=1/\sigma_i\) when \(\sigma_i\geq\tau\sigma_1\), and zero otherwise. After setting \(w=w^*(\theta)\), let \(r=M_\theta w-y\) and let \(J_\theta=\partial r/\partial\theta=U_J\operatorname{diag}(s_i)V_J^T\). The hidden-parameter update uses the truncated pseudoinverse \(\Delta\theta=-V_J\operatorname{diag}(\mathbf{1}_{s_i\geq\rho s_1}/s_i)U_J^Tr\). Small singular values are removed rather than inverted, preventing large updates along unidentifiable directions.

## Key formulas

- $$L(w,\theta)=\frac{1}{2}\left\|A\left(\Phi_\theta w\right)-y\right\|_2^2=\frac{1}{2}\left\|M_\theta w-y\right\|_2^2,\qquad M_\theta=A\Phi_\theta.$$
- $$M_\theta=U\Sigma V^T,\qquad w^*(\theta)=V\Sigma_\tau^+U^Ty,\qquad (\Sigma_\tau^+)_{ii}=\begin{cases}1/\sigma_i,&\sigma_i\geq\tau\sigma_1,\\0,&\sigma_i<\tau\sigma_1.\end{cases}$$
- $$\Delta\theta=-V_J\operatorname{diag}\left(\frac{\mathbf{1}_{s_i\geq\rho s_1}}{s_i}\right)U_J^Tr,\qquad J_\theta=U_J\operatorname{diag}(s_i)V_J^T.$$
- $$\theta_{k+1}=\theta_k+\alpha_k\Delta\theta,\qquad w_{k+1}=w^*(\theta_{k+1}).$$

## Implementation notes

Integrate this into an MLP or least-squares model with a separately accessible final linear layer. Denote all hidden weights and biases by \(\theta\), and the final layer by \(w\). At each outer iteration, first run the hidden network on a batch and collect its feature matrix \(\Phi_\theta\). For ordinary regression, \(A\) is the identity; for a least-squares neural PDE solver, construct rows for interior residuals and boundary residuals by applying the sampled differential operator to each feature. Form \(M=A\Phi_\theta\), compute a thin SVD, and replace the final-layer weights by \(w^*=V\Sigma_\tau^+U^Ty\). Use \(\tau=10^{-6}\) initially, increasing it if the condition number exceeds \(10^8\). With \(w\) fixed, compute the residual \(r=Mw-y\) and the hidden-parameter Jacobian \(J_\theta\) using automatic differentiation. For a small prototype, materialize \(J_\theta\) and compute its SVD; retain only singular values satisfying \(s_i\geq\rho s_1\), with \(\rho\) in \([10^{-5},10^{-3}]\). Form the truncated update above, use backtracking over \(\alpha\in\{1,1/2,1/4,\ldots\}\), and accept the first step that decreases the residual. The mathematical components are the exact linear solve, rank truncation, and Gauss-Newton pseudoinverse; damping, minibatch scheduling, and thresholds are empirical choices. First test a width-128 ReLU MLP on noisy 1D piecewise regression, comparing Adam, joint Gauss-Newton, and this method at equal wall-clock time. Record training and test loss, loss spikes, Jacobian condition numbers, accepted step sizes, and final accuracy. The method succeeds if it reaches lower loss faster and remains stable when hidden features become redundant; a concrete target is at least 2x lower residual at equal optimization time or equal accuracy with fewer gradient evaluations.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built and numerically verified rank-safe truncated-pseudoinverse Gauss–Newton with exact variable projection for the linear output layer. The sanity check showed full inversion produced a step norm of 1e8 along a 1e-9 singular direction, while truncation kept the step norm at 0.1; regression test MSE was 0.00374 versus Adam's 0.27034. However, default hidden Gauss–Newton steps were all rejected, so the observed gain is primarily from exact output projection and rank safety rather than successful hidden-parameter optimization.

### Mechanism check

- Verdict: Built and numerically verified rank-safe truncated-pseudoinverse Gauss–Newton with exact variable projection for the linear output layer. The sanity check showed full inversion produced a step norm of 1e8 along a 1e-9 singular direction, while truncation kept the step norm at 0.1; regression test MSE was 0.00374 versus Adam's 0.27034. However, default hidden Gauss–Newton steps were all rejected, so the observed gain is primarily from exact output projection and rank safety rather than successful hidden-parameter optimization.
- Confidence: 7/10
- Limitations: Only a small 1D tanh regression problem was tested; no PDE operator, width-128 ReLU network, repeated seeds, wall-clock-matched optimization, or joint Gauss–Newton baseline was evaluated. The hidden Jacobian was formed densely, and no speedup was established; the accuracy improvement is confounded by exact least-squares output fitting.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 8/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/1369/bench_report.json)
- [bench_vp_gn.py](https://synthcore.org/code/1369/bench_vp_gn.py)
- [rank_safe_vp_gn.py](https://synthcore.org/code/1369/rank_safe_vp_gn.py)
- [report.md](https://synthcore.org/code/1369/report.md)
- [report_bench_2026-09-04T173243.md](https://synthcore.org/code/1369/report_bench_2026-09-04T173243.md)
- [results.json](https://synthcore.org/code/1369/results.json)
- [threshold_check.py](https://synthcore.org/code/1369/threshold_check.py)
- [threshold_results.json](https://synthcore.org/code/1369/threshold_results.json)
- [Download all files as ZIP](https://synthcore.org/download/1369)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.