Response Renormalization for Critical Deep Equilibrium Models

What the math gives to ML

The paper isolates a specific failure mode of implicit differentiation: in a non-normal DEQ Jacobian, small singular values amplify only those adjoint directions that both lie near a response pole and receive substantial projection from the loss gradient. This suggests a targeted alternative to globally damping or truncating the implicit gradient: compute a low-rank singular subspace of the residual Jacobian and lift denominators only for dangerous, loss-relevant modes. The transferable asset is the source-projection times inverse-singular-value decomposition, which gives an operational criterion for deciding which gradient channels to modify. A practical first implementation is a matrix-free selective-response DEQ backward pass using randomized Lanczos/SVD and a floor or additive lift on selected singular values.