Tikhonov-Minimum-Norm Hypergradients MVP

What was built

tikhonov_experiment.py implements conjugate gradients for (H + eps I)v = b and a controlled singular quadratic bilevel problem. It compares the Tikhonov solve with the exact Moore--Penrose solution and runs a small outer optimization with fixed damping versus a decreasing damping continuation schedule.

Run:

/home/maxwelhelp/main/bin/python3 tikhonov_experiment.py

The run writes results.json.

Math checks and quantitative predictions

The Hessian has eigenvalues [0, 0.1, 1, 10]. For the stable-range test, B has zero component in the nullspace, so the prediction is (H + eps I)^(-1) B -> H^+ B, with error tending to zero. The relative error over eps=1e-1,...,1e-8 was:

0.4988, 0.09069, 0.009877, 0.0009966, 9.975e-5, 9.976e-6, 9.976e-7, 9.976e-8.

This is consistent with linear asymptotic error in epsilon.

For the eigenvalue lambda=0.1 component, the exact prediction is relative bias eps/(0.1+eps). At eps=1e-8, observed bias was 9.9999988e-8 versus predicted 9.9999990e-8.

For a deliberate violation of the range condition, B was given nullspace component 0.8. The nullspace component of the regularized derivative is exactly 0.8/eps; the measured log-log slope of solution norm versus epsilon was -0.99999993 (predicted -1). The quantity eps * null_component stayed approximately 0.8 throughout the sweep.

Thus all three mechanism predictions manifested numerically.

Mini-experiment

The inner objective is a singular quadratic and the outer objective is squared distance to a target. The exact minimum-norm hypergradient at the initial point was 203.2592. A fixed eps=0.1 outer run ended with loss 0.11922; the continuation run (epsilon decaying from 0.1 to a floor of 1e-7) ended with loss 0.12423. Therefore continuation did not beat fixed damping in this tiny, well-conditioned-on-the-range problem.

Verdict

The proposed mathematical mechanism is real: Tikhonov derivatives converge to the pseudoinverse derivative under the stable-range condition, with the predicted first-order damping bias, while a nullspace mixed derivative produces the predicted 1/eps instability. The MVP does not establish a practical optimization win; in the small outer test, fixed damping was slightly better.

Limitations

No MNIST/MLP experiment, nonsmooth generalized Hessian experiment, noisy hypergradient variance study, wall-clock benchmark, or large-width scaling study was performed. The quadratic test uses exact matrix products for the reported outer comparison, although the Tikhonov linear systems are also solved by the included CG implementation in the core sweep.