Non-Negative Conjugate Gradients

What the math gives to ML

The paper turns conjugate gradients into a solver for strictly convex quadratic programs with non-negativity and optional linear equality constraints by combining matrix-free Krylov solves with a primal-dual active-set loop. The transferable asset is the separation between cheap inner linear algebra and finite outer sign decisions: variables that violate positivity are removed from the free set, while bound variables with negative reduced gradient are reactivated. This is especially suitable for neural components whose training subproblem is a nonnegative or simplex-constrained least-squares problem, such as mixture-of-experts routers, nonnegative dictionary codes, and constrained linear heads. The first practical target should be a router or adapter solved repeatedly with matrix-vector products rather than a generic optimizer replacement for all network parameters.