Structure-Guided Gauss-Newton Method: Linear Advection-Reaction Equation

What the math gives to ML

The paper's transferable contribution is a variable-projection training scheme for networks that are linear in output parameters and nonlinear in hidden parameters. The output coefficients are solved to least-squares optimality at every outer iteration, while only the hidden parameters receive a Gauss-Newton update. The practically important addition is explicit removal of singular or nearly singular directions, which can prevent redundant features, inactive ReLU units, and ill-conditioned Jacobians from causing unstable parameter steps.