{
 "artifacts": null,
 "category": "optimization",
 "description": "Modify implicit differentiation in a DEQ by lifting inverse-response denominators only for singular directions that are simultaneously ill-conditioned and strongly excited by the loss gradient. Leave all well-conditioned directions unchanged, preserving useful sensitivity instead of applying global gradient clipping or full Tikhonov damping.",
 "formulas_latex": [
  "$$K^{\\top}v=g,\\qquad K=I-\\partial_{z}f_{\\theta}(z^{\\star},x),\\qquad g=\\nabla_{z^{\\star}}\\mathcal{L}.$$",
  "$$K=U\\Sigma V^{\\top},\\qquad v=U\\Sigma^{-1}V^{\\top}g=\\sum_i\\frac{v_i^{\\top}g}{\\sigma_i}u_i.$$",
  "$$C=\\left\\{i:\\sigma_i\u003c\\tau_\\sigma\\ \\land\\ \\frac{|v_i^{\\top}g|}{\\|g\\|_2+\\varepsilon}\u003e\\tau_g\\right\\},\\qquad \\widetilde{\\sigma}_i=\\begin{cases}\\max(\\sigma_i,\\tau_\\sigma),\u0026i\\in C,\\\\\\sigma_i,\u0026i\\notin C,\\end{cases}$$",
  "$$\\widetilde v=\\sum_i\\frac{v_i^{\\top}g}{\\widetilde{\\sigma}_i}u_i.$$"
 ],
 "id": 108,
 "implementation": "Integrate this into the backward pass of a DEQ or another implicit layer, immediately after the forward equilibrium state $z^\\star$ has been found and before parameter gradients are computed. The tensor to modify is the adjoint vector passed into vector-Jacobian products with $f_\\theta(z^\\star,x)$; do not alter the forward fixed-point solver in the first experiment. Use a matrix-free randomized SVD or Lanczos routine because explicitly materializing $K$ is infeasible for large hidden states. Pseudocode: (1) compute $z^\\star$ with the existing fixed-point solver and compute $g=\\nabla_z\\mathcal L$; (2) define JVP/VJP routines for $Kq=q-J_fq$ and $K^\\top q=q-J_f^\\top q$; (3) estimate the smallest $r$ singular triplets $(\\sigma_i,u_i,v_i)$ using Lanczos on $K^\\top K$ or a randomized eigensolver; (4) calculate $a_i=v_i^\\top g$; (5) mark mode $i$ critical when $\\sigma_i\u003c\\tau_\\sigma$ and $|a_i|/(\\|g\\|_2+\\varepsilon)\u003e\\tau_g$; (6) form the low-rank corrected adjoint $\\widetilde v=v_{\\mathrm{base}}+\\sum_{i\\in C}a_i(1/\\widetilde\\sigma_i-1/\\sigma_i)u_i$, where $v_{\\mathrm{base}}$ is the ordinary iterative solve or the truncated SVD response; (7) feed $\\widetilde v$ into the standard DEQ parameter-gradient VJP. Estimate singular values and vectors empirically; the paper supplies the response decomposition, while $\\tau_\\sigma$, $\\tau_g$, rank $r$, and $\\lambda_0$ are hyperparameters. Start with $\\tau_\\sigma$ equal to the 1st-5th percentile of estimated singular values and sweep $\\tau_g\\in\\{0.01,0.05,0.1\\}$. The first cheap experiment should use a 2-4 layer MLP DEQ on MNIST or CIFAR-10, comparing exact/standard implicit differentiation, global gradient clipping, Tikhonov damping, and selective renormalization at equal training steps. Measure loss variance, gradient norm spikes, failed linear solves, final accuracy, and wall-clock overhead. Success is fewer adjoint-solve failures and lower gradient variance with no more than 1-2% accuracy loss; on deliberately near-critical DEQs, target faster loss descent and substantially fewer exploding updates at equal FLOPs.",
 "math_summary": "The paper defines the DEQ residual Jacobian as $K=I-\\partial_z f_\\theta(z^\\star,x)$, the equilibrium loss gradient as $g=\\nabla_{z^\\star}\\mathcal{L}$, and the implicit adjoint $v$ by $K^\\top v=g$. For the singular-value decomposition $K=U\\Sigma V^\\top$, where $u_i$ and $v_i$ are left and right singular vectors and $\\sigma_i\u003e0$ are singular values, the response is $v=\\sum_i (v_i^\\top g/\\sigma_i)u_i$. The scalar $a_i=v_i^\\top g$ is the loss-source projection and $1/\\sigma_i$ is the amplification factor. Use a critical set $C=\\{i:\\sigma_i\u003c\\tau_\\sigma\\ \\mathrm{and}\\ |a_i|/\\|g\\|_2\u003e\\tau_g\\}$ and replace only its denominators by $\\widetilde{\\sigma}_i=\\max(\\sigma_i,\\tau_\\sigma)$ or, more smoothly, $\\widetilde{\\sigma}_i=\\sigma_i+\\lambda_i$ with $\\lambda_i=\\lambda_0\\mathbf{1}_{i\\in C}$. The renormalized adjoint is $\\widetilde v=\\sum_i (a_i/\\widetilde{\\sigma}_i)u_i$. Here $\\tau_\\sigma$ is a singular-value safety threshold, $\\tau_g$ is a normalized source threshold, and $\\lambda_0$ controls the maximum response reduction. This is a backward-response surrogate unless the forward residual is explicitly changed; it should therefore be evaluated as a gradient stabilization method rather than assumed to be the exact gradient of the original equilibrium objective.",
 "math_tags": [
  "linear-algebra",
  "dynamical-systems",
  "optimization",
  "numerical-analysis"
 ],
 "ml_areas": [
  "optimizer",
  "training",
  "ssm"
 ],
 "paper": {
  "arxiv_id": "2608.23725",
  "arxiv_url": "https://arxiv.org/abs/2608.23725",
  "summary_what_math_gives_to_ml": "The paper isolates a specific failure mode of implicit differentiation: in a non-normal DEQ Jacobian, small singular values amplify only those adjoint directions that both lie near a response pole and receive substantial projection from the loss gradient. This suggests a targeted alternative to globally damping or truncating the implicit gradient: compute a low-rank singular subspace of the residual Jacobian and lift denominators only for dangerous, loss-relevant modes. The transferable asset is the source-projection times inverse-singular-value decomposition, which gives an operational criterion for deciding which gradient channels to modify. A practical first implementation is a matrix-free selective-response DEQ backward pass using randomized Lanczos/SVD and a floor or additive lift on selected singular values.",
  "title": "Response Renormalization for Critical Deep Equilibrium Models",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 6,
  "novelty": 7,
  "usefulness": 7
 },
 "solves": [
  "stability",
  "accuracy",
  "speedup"
 ],
 "title": "Selective Adjoint Response Renormalization",
 "url": "https://synthcore.org/idea/108/selective-adjoint-response-renormalization",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}