✓✓ Beats tuned baseline 2026

Rank-Safe Variable-Projection Gauss-Newton

Implementation & benchmark of arXiv:2607.07506 — Structure-Guided Gauss-Newton Method: Linear Advection-Reaction Equation

Usefulness6/10

Difficulty6/10

Novelty5/10

Optimization Speedup Stability Accuracy Optimizer Mlp Training dynamics Loss Optimization Linear algebra Numerical analysis

Source paper: Structure-Guided Gauss-Newton Method: Linear Advection-Reaction Equation arXiv:2607.07506 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Separate a neural network into nonlinear hidden parameters and a linear output layer. Solve the output layer exactly by least squares, then update hidden parameters with a truncated-pseudoinverse Gauss-Newton step that discards numerically singular directions.

Formulas

$$L(w,\theta)=\frac{1}{2}\left\|A\left(\Phi_\theta w\right)-y\right\|_2^2=\frac{1}{2}\left\|M_\theta w-y\right\|_2^2,\qquad M_\theta=A\Phi_\theta.$$

$$M_\theta=U\Sigma V^T,\qquad w^*(\theta)=V\Sigma_\tau^+U^Ty,\qquad (\Sigma_\tau^+)_{ii}=\begin{cases}1/\sigma_i,&\sigma_i\geq\tau\sigma_1,\\0,&\sigma_i<\tau\sigma_1.\end{cases}$$

$$\Delta\theta=-V_J\operatorname{diag}\left(\frac{\mathbf{1}_{s_i\geq\rho s_1}}{s_i}\right)U_J^Tr,\qquad J_\theta=U_J\operatorname{diag}(s_i)V_J^T.$$

$$\theta_{k+1}=\theta_k+\alpha_k\Delta\theta,\qquad w_{k+1}=w^*(\theta_{k+1}).$$

Mathematical statement

Let $\phi_\theta(x)\in\mathbb{R}^{m}$ be the feature vector produced by hidden parameters $\theta$, let $w\in\mathbb{R}^{m\times q}$ be the linear output matrix, and let $A$ denote the sampled observation, differential, or residual operator. With stacked targets $y$, define $\Phi_\theta$ as the feature matrix and $M_\theta=A\Phi_\theta$. The least-squares objective is $L(w,\theta)=\frac{1}{2}\|M_\theta w-y\|_2^2$. For fixed $\theta$, compute the variable-projection solution using the singular value decomposition $M_\theta=U\Sigma V^T$: $w^*(\theta)=V\Sigma_\tau^+U^Ty$, where $(\Sigma_\tau^+)_{ii}=1/\sigma_i$ when $\sigma_i\geq\tau\sigma_1$, and zero otherwise. After setting $w=w^*(\theta)$, let $r=M_\theta w-y$ and let $J_\theta=\partial r/\partial\theta=U_J\operatorname{diag}(s_i)V_J^T$. The hidden-parameter update uses the truncated pseudoinverse $\Delta\theta=-V_J\operatorname{diag}(\mathbf{1}_{s_i\geq\rho s_1}/s_i)U_J^Tr$. Small singular values are removed rather than inverted, preventing large updates along unidentifiable directions.

Implementation notes

Integrate this into an MLP or least-squares model with a separately accessible final linear layer. Denote all hidden weights and biases by $\theta$, and the final layer by $w$. At each outer iteration, first run the hidden network on a batch and collect its feature matrix $\Phi_\theta$. For ordinary regression, $A$ is the identity; for a least-squares neural PDE solver, construct rows for interior residuals and boundary residuals by applying the sampled differential operator to each feature. Form $M=A\Phi_\theta$, compute a thin SVD, and replace the final-layer weights by $w^*=V\Sigma_\tau^+U^Ty$. Use $\tau=10^{-6}$ initially, increasing it if the condition number exceeds $10^8$. With $w$ fixed, compute the residual $r=Mw-y$ and the hidden-parameter Jacobian $J_\theta$ using automatic differentiation. For a small prototype, materialize $J_\theta$ and compute its SVD; retain only singular values satisfying $s_i\geq\rho s_1$, with $\rho$ in $[10^{-5},10^{-3}]$. Form the truncated update above, use backtracking over $\alpha\in\{1,1/2,1/4,\ldots\}$, and accept the first step that decreases the residual. The mathematical components are the exact linear solve, rank truncation, and Gauss-Newton pseudoinverse; damping, minibatch scheduling, and thresholds are empirical choices. First test a width-128 ReLU MLP on noisy 1D piecewise regression, comparing Adam, joint Gauss-Newton, and this method at equal wall-clock time. Record training and test loss, loss spikes, Jacobian condition numbers, accepted step sizes, and final accuracy. The method succeeds if it reaches lower loss faster and remains stable when hidden features become redundant; a concrete target is at least 2x lower residual at equal optimization time or equal accuracy with fewer gradient evaluations.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

How to read this: The implementation beat the baseline, but the claimed mechanism did not reproduce at benchmark scale. The gain may come from another effect; treat the cause as unresolved.

Stage 1 — Mechanism check agent confidence 7/10

Built and numerically verified rank-safe truncated-pseudoinverse Gauss–Newton with exact variable projection for the linear output layer. The sanity check showed full inversion produced a step norm of 1e8 along a 1e-9 singular direction, while truncation kept the step norm at 0.1; regression test MSE was 0.00374 versus Adam's 0.27034. However, default hidden Gauss–Newton steps were all rejected, so the observed gain is primarily from exact output projection and rank safety rather than successful hidden-parameter optimization.

Agent confidence: 7/10

Baseline: Adam: train loss 0.13666, test MSE 0.27034, runtime 2.42 s
Idea: Rank-safe VP-GN: train loss 0.001881, test MSE 0.003736, runtime 9.45 s, 0 accepted hidden steps, maximum Jacobian condition number 2.82e18
Sanity: Full pseudoinverse step norm 1.0e8 versus truncated step norm 0.1; truncated rank 1
Threshold check: rho=1e-5, 1e-4, and 1e-3 accepted 0/20 steps; rho=1e-2 accepted 20/20 with essentially unchanged test MSE 0.003754

Limitations: Only a small 1D tanh regression problem was tested; no PDE operator, width-128 ReLU network, repeated seeds, wall-clock-matched optimization, or joint Gauss–Newton baseline was evaluated. The hidden Jacobian was formed densely, and no speedup was established; the accuracy improvement is confounded by exact least-squares output fitting.

How to run: python3 rank_safe_vp_gn.py && /home/maxwelhelp/main/bin/python3 threshold_check.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.07506) Sep 4, 2026

Beats the tuned baseline by 25.7% (p=0.0081, wins 8 / 8 paired seeds; metric: lower is better).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 11.6078
Idea mean: 8.6234
Effect (Δ): -2.9844 (−25.7%; negative = idea better)
Wins: 8 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±13.4%
Mechanism: Not confirmed ✗
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched tabular/Friedman regression track was tested. The dense hidden Jacobian was limited to the tiny MLP and 400-sample subsets; wall-clock speed was not the primary comparison, no vision/sequence/dynamics tracks were run, and the trained-model signature did not reproduce the toy singular-direction amplification.

How to run: python3 bench_vp_gn.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 4, 2026 latest Worked ✓

Agent confidence: 8/10

Built Rank-Safe Variable-Projection Gauss–Newton as a local modification of the tabular mlp_tiny training procedure, using exact SVD output projection, truncated hidden Jacobian pseudoinverse, and backtracking. Against an Adam baseline tuned over the same learning-rate union, the idea achieved lower test MSE on all 8 paired seeds: mean 8.6234 versus 11.6078, paired delta -2.9844, permutation p=0.0081, so the harness verdict is idea better (significant). The neural-scale mechanism signature was not confirmed: observed full/truncated step amplification was 1.0 rather than the toy prediction, and the idea was not shown to be faster.

Baseline: mlp_tiny Adam, best lr=0.01; 8-seed test MSE mean 11.607812, std 1.173520; per-seed [10.810469, 12.079232, 10.855374, 10.777785, 9.649698, 12.845276, 12.719051, 13.125610]
Idea: Rank-safe VP-GN, best shared-grid lr=0.001; 8-seed test MSE mean 8.623417, std 1.450602; per-seed [7.800422, 11.718672, 10.022994, 6.932705, 7.960086, 8.643535, 8.414509, 7.494410]
Mechanism signature: Predicted amplification 1.000000064e9; trained-model observed amplification 1.0, confirmed=false
Paired delta: -2.984396; idea wins 8/8 pairs; permutation p=0.0081
Sanity: Toy singular-direction check: full step norm 1.00000008e8, truncated step norm 0.1, kept rank 1

Benchmark result