A Fixed-Penalty Linearized Augmented Lagrangian Method with Classical Multiplier Updates

What the math gives to ML

The paper provides a practical equality-constrained optimizer that avoids solving a nonlinear augmented-Lagrangian subproblem: linearize the constraint, add a proximal or Gauss–Newton quadratic, solve one symmetric positive-definite system, and then perform the classical nonlinear residual multiplier update. The transferable asset is the deliberate separation between the linearized primal step and the exact nonlinear constraint residual, together with a fixed penalty parameter and local control of the resulting quadratic error. This is useful for neural training problems with explicit equality constraints, such as orthogonality, normalization, conservation, or expert-load balancing, where ordinary penalty losses require difficult penalty scheduling. The most direct experiment is a constrained optimizer implemented with conjugate gradients and compared against AdamW plus a tuned quadratic penalty.