{
 "artifacts": null,
 "category": "training",
 "description": "Replace the usual inverse-Hessian implicit hypergradient with the derivative of the minimum-norm inner solution. Compute it as the limit of derivatives of a uniquely solvable Tikhonov-regularized problem, using a decreasing damping parameter and conjugate-gradient solves. This should make bilevel training usable when the inner model is overparameterized or has flat directions.",
 "formulas_latex": [
  "$$\\min_{\\theta\\in\\Theta}\\mathcal{L}(x^{\\star}(\\theta),\\theta),\\qquad x^{\\star}(\\theta)\\in\\mathcal S(\\theta):=\\operatorname*{argmin}_{x\\in\\mathbb R^n}f(x,\\theta).$$",
  "$$x_\\varepsilon(\\theta):=\\operatorname*{argmin}_{x\\in\\mathbb R^n}\\left[f(x,\\theta)+\\frac{\\varepsilon}{2}\\|x\\|_2^2\\right],\\qquad \\varepsilon\u003e0.$$",
  "$$(\\nabla^2_{xx}f(x_\\varepsilon,\\theta)+\\varepsilon I)D_\\theta x_\\varepsilon=-\\nabla^2_{x\\theta}f(x_\\varepsilon,\\theta),\\qquad D_\\theta x_\\varepsilon\\xrightarrow[\\varepsilon\\downarrow0]{}-H^+B.$$",
  "$$\\frac{\\mathrm d}{\\mathrm dt}F(\\mu(t))=V\\dot{\\mu}(t)\\qquad\\forall V\\in J_F(\\mu(t)),\\quad\\text{for almost every }t.$$"
 ],
 "id": 2796,
 "implementation": "(1) Integration point: use this as the backward pass through an inner training or optimization layer. The inner variable \\(x\\) can be a small model's weights, a feature extractor, a differentiable data-cleaning vector, or learned regularization parameters; \\(\\theta\\) is the outer hyperparameter or data-perturbation vector. Do not differentiate through every inner optimizer step. Instead, solve the inner problem, add a temporary quadratic damping term, and solve one linear system for the vector-Jacobian product needed by the outer loss. (2) Pseudocode: `x = inner_solve(f(x, theta))`; choose `eps = eps0 * decay**outer_step` but clamp it to `eps_min`; define the Hessian-vector product `hvp(v) = grad_x(dot(grad_x(f(x,theta)), v))`; define `A(v) = hvp(v) + eps*v`; compute `b = grad_x(L_outer(x,theta))`; solve `A(v) = b` with conjugate gradients; return hypergradient `g_theta = grad_theta(L_outer(x,theta)) - grad_theta(dot(grad_x(f(x,theta)), stop_gradient(v)))`. This is the adjoint form of \\((H+\\varepsilon I)D_\\theta x_\\varepsilon=-B\\). If the inner objective is nonsmooth, use the autodiff generalized-Hessian-vector product produced by the selected subgradient and repeat with several damping values. (3) Computed from the mathematics: the positive-definite damped operator \\(H+\\varepsilon I\\), its stable linear solve, and the continuation limit as damping decreases. Estimated empirically: the damping schedule, CG tolerance, and whether the solution has stabilized; monitor relative changes in \\(v_\\varepsilon\\) for \\(\\varepsilon,\\varepsilon/2\\). A practical stopping rule is `||v_eps - v_eps/2|| / (||v_eps||+1e-8) \u003c 0.05`. The method assumes the mixed derivative lies in the stable range of the generalized Hessian; detect violations through CG residuals and exploding successive estimates. (4) First experiment: use a two-layer overparameterized MLP on MNIST or CIFAR-10, with the inner objective being training cross-entropy plus a tunable data-weight or weight-decay vector and the outer objective validation cross-entropy. Compare unrolled differentiation, naive damped inverse-Hessian differentiation, and this continuation method at equal outer iterations. Measure validation loss, hypergradient cosine similarity against a long unrolled reference, CG cost, and maximum gradient norm. Success means substantially lower hypergradient variance and no divergence as width increases or weight decay approaches zero, while achieving lower validation loss at the same number of outer updates.",
 "math_summary": "The bilevel problem is the outer objective \\(\\min_{\\theta\\in\\Theta}\\mathcal L(x^\\star(\\theta),\\theta)\\), where \\(x^\\star(\\theta)\\in\\arg\\min_x f(x,\\theta)\\). When the solution set is non-singleton, select the canonical solution \\(x^\\dagger(\\theta):=\\operatorname*{argmin}_{x\\in\\arg\\min_z f(z,\\theta)}\\|x\\|_2\\). For \\(\\varepsilon\u003e0\\), Tikhonov regularization defines the unique damped solution \\(x_\\varepsilon(\\theta):=\\arg\\min_x[f(x,\\theta)+\\frac{\\varepsilon}{2}\\|x\\|_2^2]\\). If \\(H=\\nabla^2_{xx}f(x_\\varepsilon,\\theta)\\) and \\(B=\\nabla^2_{x\\theta}f(x_\\varepsilon,\\theta)\\), differentiating the regularized optimality condition gives \\((H+\\varepsilon I)D_\\theta x_\\varepsilon=-B\\). As \\(\\varepsilon\\downarrow0\\), this converges, under the paper's uniform-range and eigenvalue-control assumptions, to the minimum-norm generalized derivative; in the linear fixed-point case the limit is \\(D_\\theta x^\\dagger=-H^+B\\), where \\(H^+\\) is the Moore--Penrose pseudoinverse. For nonsmooth \\(f\\), replace \\(H\\) by an admissible generalized Hessian element and use the conservative-Jacobian property: for an absolutely continuous path \\(\\mu\\), \\(\\frac{d}{dt}F(\\mu(t))=V\\dot\\mu(t)\\) for every \\(V\\in J_F(\\mu(t))\\), with \\(J_F\\) locally bounded and having a closed graph.",
 "math_tags": [
  "optimization",
  "convex-analysis",
  "linear-algebra",
  "dynamical-systems"
 ],
 "ml_areas": [
  "optimizer",
  "training",
  "loss",
  "fine-tuning"
 ],
 "paper": {
  "arxiv_id": "2608.28899",
  "arxiv_url": "https://arxiv.org/abs/2608.28899",
  "summary_what_math_gives_to_ml": "The paper addresses a failure mode of standard implicit differentiation: an inner optimization problem may have multiple minimizers, so its Jacobian is singular even though a stable, canonical solution can be selected. Its transferable asset is the minimal-norm selection together with limiting Tikhonov regularization, which replaces an unstable inverse by a controlled pseudoinverse-like solve and extends differentiation to nonsmooth objectives through conservative Jacobians. A practical neural-network use is a differentiable optimization layer for hyperparameter tuning, meta-learning, or data reweighting that returns the minimal-norm inner solution and computes hypergradients with damped linear solves rather than blindly inverting a singular Hessian.",
  "title": "Differentiating Minimal-Norm Solutions to Parametric Optimization Problems",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 6,
  "usefulness": 7
 },
 "solves": [
  "stability",
  "accuracy",
  "sample-efficiency"
 ],
 "title": "Tikhonov-Minimum-Norm Hypergradients",
 "url": "https://synthcore.org/idea/2796/tikhonov-minimum-norm-hypergradients",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)"
 }
}