Tikhonov-Minimum-Norm Hypergradients

(1) Integration point: use this as the backward pass through an inner training or optimization layer. The inner variable \(x\) can be a small model's weights, a feature extractor, a differentiable data-cleaning vector, or learned regularization parameters; \(\theta\) is the outer hyperparameter or data-perturbation vector. Do not differentiate through every inner optimizer step. Instead, solve the inner problem, add a temporary quadratic damping term, and solve one linear system for the vector-Jacobian product needed by the outer loss. (2) Pseudocode: x = inner_solve(f(x, theta)); choose eps = eps0 * decay**outer_step but clamp it to eps_min; define the Hessian-vector product hvp(v) = grad_x(dot(grad_x(f(x,theta)), v)); define A(v) = hvp(v) + eps*v; compute b = grad_x(L_outer(x,theta)); solve A(v) = b with conjugate gradients; return hypergradient g_theta = grad_theta(L_outer(x,theta)) - grad_theta(dot(grad_x(f(x,theta)), stop_gradient(v))). This is the adjoint form of \((H+\varepsilon I)D_\theta x_\varepsilon=-B\). If the inner objective is nonsmooth, use the autodiff generalized-Hessian-vector product produced by the selected subgradient and repeat with several damping values. (3) Computed from the mathematics: the positive-definite damped operator \(H+\varepsilon I\), its stable linear solve, and the continuation limit as damping decreases. Estimated empirically: the damping schedule, CG tolerance, and whether the solution has stabilized; monitor relative changes in \(v_\varepsilon\) for \(\varepsilon,\varepsilon/2\). A practical stopping rule is ||v_eps - v_eps/2|| / (||v_eps||+1e-8) < 0.05. The method assumes the mixed derivative lies in the stable range of the generalized Hessian; detect violations through CG residuals and exploding successive estimates. (4) First experiment: use a two-layer overparameterized MLP on MNIST or CIFAR-10, with the inner objective being training cross-entropy plus a tunable data-weight or weight-decay vector and the outer objective validation cross-entropy. Compare unrolled differentiation, naive damped inverse-Hessian differentiation, and this continuation method at equal outer iterations. Measure validation loss, hypergradient cosine similarity against a long unrolled reference, CG cost, and maximum gradient norm. Success means substantially lower hypergradient variance and no divergence as width increases or weight decay approaches zero, while achieving lower validation loss at the same number of outer updates.