Augmented KKT preconditioner for constrained neural training

Integrate this at the optimizer step for a model with explicit equality constraints, not inside the forward pass. Examples include enforcing conservation laws in a physics-informed network, preserving selected outputs during continual learning, or imposing adapter orthogonality. At iterate $\theta$, compute the constraint residual $r=c(\theta)$ and expose Jacobian-vector products for $J=\partial c/\partial\theta$. Form a damped curvature operator $H=G+\lambda I$, where $G$ is a Gauss--Newton, empirical-Fisher, or diagonal-curvature approximation and $\lambda>0$ is damping. Choose $M_c=\operatorname{diag}(\operatorname{mean}(J_i^2)+\epsilon)$ or a small constraint Gram matrix. Solve the adapted KKT system approximately with MINRES using block diagonal preconditioner $P=\operatorname{diag}(P_\theta,P_c)$, where $P_\theta\approx H+\eta J^TM_c^{-1}J$ and $P_c\approx JP_\theta^{-1}J^T$. Pseudocode: (1) evaluate $r$ and the loss gradient $g$; (2) define matrix-free products $v\mapsto Hv$, $v\mapsto Jv$, and $u\mapsto J^Tu$; (3) run MINRES on $K_\eta[\Delta\theta,\lambda]=[-g,-r]$; (4) update $\theta\leftarrow\theta+\alpha\Delta\theta$ and the multiplier estimate $\mu\leftarrow\mu+\alpha\lambda$; (5) refresh curvature and Jacobian statistics periodically. The paper's algebra supplies the augmentation and Schur structure; estimate $H$, $M_c$, and the Schur approximation empirically. Sweep $\eta\in\{0.1,1,10,100\}$ and monitor the constraint-conditioned spectrum or short Lanczos estimates. First test on a two-layer MLP solving constrained regression with 10--100 constraints, comparing Adam plus a penalty term and an unpreconditioned KKT solve at equal Jacobian-vector-product count. Success means fewer Krylov iterations, lower constraint violation at equal loss, and stable behavior as parameter count or constraint scaling increases.