✗ Failed on benchmark 2026

Tikhonov-Minimum-Norm Hypergradients

Implementation & benchmark of arXiv:2608.28899 — Differentiating Minimal-Norm Solutions to Parametric Optimization Problems

Usefulness7/10

Difficulty5/10

Novelty6/10

Training Stability Accuracy Sample efficiency Optimizer Training Loss Fine tuning Optimization Convex analysis Linear algebra Dynamical systems

Source paper: Differentiating Minimal-Norm Solutions to Parametric Optimization Problems arXiv:2608.28899 ⓘ · analyzed Sep 1, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace the usual inverse-Hessian implicit hypergradient with the derivative of the minimum-norm inner solution. Compute it as the limit of derivatives of a uniquely solvable Tikhonov-regularized problem, using a decreasing damping parameter and conjugate-gradient solves. This should make bilevel training usable when the inner model is overparameterized or has flat directions.

Formulas

$$\min_{\theta\in\Theta}\mathcal{L}(x^{\star}(\theta),\theta),\qquad x^{\star}(\theta)\in\mathcal S(\theta):=\operatorname*{argmin}_{x\in\mathbb R^n}f(x,\theta).$$

$$x_\varepsilon(\theta):=\operatorname*{argmin}_{x\in\mathbb R^n}\left[f(x,\theta)+\frac{\varepsilon}{2}\|x\|_2^2\right],\qquad \varepsilon>0.$$

$$(\nabla^2_{xx}f(x_\varepsilon,\theta)+\varepsilon I)D_\theta x_\varepsilon=-\nabla^2_{x\theta}f(x_\varepsilon,\theta),\qquad D_\theta x_\varepsilon\xrightarrow[\varepsilon\downarrow0]{}-H^+B.$$

$$\frac{\mathrm d}{\mathrm dt}F(\mu(t))=V\dot{\mu}(t)\qquad\forall V\in J_F(\mu(t)),\quad\text{for almost every }t.$$

Mathematical statement

The bilevel problem is the outer objective $\min_{\theta\in\Theta}\mathcal L(x^\star(\theta),\theta)$, where $x^\star(\theta)\in\arg\min_x f(x,\theta)$. When the solution set is non-singleton, select the canonical solution $x^\dagger(\theta):=\operatorname*{argmin}_{x\in\arg\min_z f(z,\theta)}\|x\|_2$. For $\varepsilon>0$, Tikhonov regularization defines the unique damped solution $x_\varepsilon(\theta):=\arg\min_x[f(x,\theta)+\frac{\varepsilon}{2}\|x\|_2^2]$. If $H=\nabla^2_{xx}f(x_\varepsilon,\theta)$ and $B=\nabla^2_{x\theta}f(x_\varepsilon,\theta)$, differentiating the regularized optimality condition gives $(H+\varepsilon I)D_\theta x_\varepsilon=-B$. As $\varepsilon\downarrow0$, this converges, under the paper's uniform-range and eigenvalue-control assumptions, to the minimum-norm generalized derivative; in the linear fixed-point case the limit is $D_\theta x^\dagger=-H^+B$, where $H^+$ is the Moore--Penrose pseudoinverse. For nonsmooth $f$, replace $H$ by an admissible generalized Hessian element and use the conservative-Jacobian property: for an absolutely continuous path $\mu$, $\frac{d}{dt}F(\mu(t))=V\dot\mu(t)$ for every $V\in J_F(\mu(t))$, with $J_F$ locally bounded and having a closed graph.

Implementation notes

(1) Integration point: use this as the backward pass through an inner training or optimization layer. The inner variable $x$ can be a small model's weights, a feature extractor, a differentiable data-cleaning vector, or learned regularization parameters; $\theta$ is the outer hyperparameter or data-perturbation vector. Do not differentiate through every inner optimizer step. Instead, solve the inner problem, add a temporary quadratic damping term, and solve one linear system for the vector-Jacobian product needed by the outer loss. (2) Pseudocode: x = inner_solve(f(x, theta)); choose eps = eps0 * decay**outer_step but clamp it to eps_min; define the Hessian-vector product hvp(v) = grad_x(dot(grad_x(f(x,theta)), v)); define A(v) = hvp(v) + eps*v; compute b = grad_x(L_outer(x,theta)); solve A(v) = b with conjugate gradients; return hypergradient g_theta = grad_theta(L_outer(x,theta)) - grad_theta(dot(grad_x(f(x,theta)), stop_gradient(v))). This is the adjoint form of $(H+\varepsilon I)D_\theta x_\varepsilon=-B$. If the inner objective is nonsmooth, use the autodiff generalized-Hessian-vector product produced by the selected subgradient and repeat with several damping values. (3) Computed from the mathematics: the positive-definite damped operator $H+\varepsilon I$, its stable linear solve, and the continuation limit as damping decreases. Estimated empirically: the damping schedule, CG tolerance, and whether the solution has stabilized; monitor relative changes in $v_\varepsilon$ for $\varepsilon,\varepsilon/2$. A practical stopping rule is ||v_eps - v_eps/2|| / (||v_eps||+1e-8) < 0.05. The method assumes the mixed derivative lies in the stable range of the generalized Hessian; detect violations through CG residuals and exploding successive estimates. (4) First experiment: use a two-layer overparameterized MLP on MNIST or CIFAR-10, with the inner objective being training cross-entropy plus a tunable data-weight or weight-decay vector and the outer objective validation cross-entropy. Compare unrolled differentiation, naive damped inverse-Hessian differentiation, and this continuation method at equal outer iterations. Measure validation loss, hypergradient cosine similarity against a long unrolled reference, CG cost, and maximum gradient norm. Success means substantially lower hypergradient variance and no divergence as width increases or weight decay approaches zero, while achieving lower validation loss at the same number of outer updates.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a readable Tikhonov/CG MVP for a singular quadratic bilevel problem. All three mechanism predictions were confirmed: stable-range convergence to H^+B, exact O(epsilon) component bias, and epsilon^-1 divergence under nullspace-range violation. The practical outer comparison was not a win: fixed damping ended at loss 0.11922 versus 0.12423 for continuation, so this supports the math but not improved optimization performance in this toy.

Agent confidence: 9/10

Baseline: Fixed damping eps=0.1: final outer loss 0.1192212364; exact initial hypergradient 203.2592.
Idea: Continuation eps=0.1*0.7^step clipped at 1e-7: final outer loss 0.1242258204. Stable-limit relative error at eps=1e-8 was 9.98e-8; lambda=0.1 bias observed 9.9999988e-8 vs predicted 9.9999990e-8; unstable nullspace log-log slope -0.99999993 vs predicted -1.

Limitations: Only a 4-dimensional synthetic quadratic was tested. No MNIST/MLP, nonsmooth generalized-Hessian case, noisy hypergradient variance, wall-clock benchmark, GPU experiment, or width-scaling study was performed.

How to run: python3 tikhonov_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.28899) Sep 2, 2026

Significantly WORSE than the tuned baseline (+2.8%, p=0.0403, wins 1 / 8 paired seeds).

Benchmark: Tabular regression (Friedman)
Model: mlp_tiny
Paired seeds: 8
Baseline mean: 9.6423
Idea mean: 9.9093
Effect (Δ): 0.267 (+2.8%; negative = idea better)
Wins: 1 / 8 paired seeds
p-value: 0.0403 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±3.1%
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 9 configs

Limitations:

The fixed built-in tracks do not expose a true outer hyperparameter optimization problem, so the practical intervention was represented as Tikhonov continuation during ordinary MLP training rather than an end-to-end implicit hypergradient through an inner/outer split. No vision, sequence, or dynamics tracks were run because tabular is the prescribed structural match for optimizer/regularizer ideas; no width scaling, unrolled-reference cosine study, wall-clock comparison, or nonsmooth generalized-Hessian case was tested.

How to run: python3 bench_tikhonov.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 9/10

Built a local tabular benchmark modification using the shared mlp_tiny architecture, identical data, epochs, batch size, and learning-rate union; baseline Adam included a weight-decay sweep, while the idea used decreasing Tikhonov damping. On 8 paired seeds, baseline MSE was 9.6423 and the best idea MSE was 9.9093, delta +0.2670 with permutation p=0.04035, therefore the idea was significantly worse. The trained-model CG signature was also not confirmed: observed maximum residual 80.76 versus predicted tolerance 0.000507.

Baseline: Best cfg lr=0.006, wd=0.001; full 8-seed test MSE mean=9.6423278451, std=1.0473264846; per-seed=[9.8919839859,10.5964794159,8.6700820923,9.0212182999,7.6159310341,10.5873336792,9.9360847473,10.8195095062].
Idea: Best cfg lr=0.006, eps0=0.01, eps_min=1e-6, decay=0.7; full 8-seed test MSE mean=9.9093276858, std=1.2741297958; delta_mean=+0.2669998407, idea_wins=1/8, permutation p=0.04035. Signature observed_max_residual=80.7649307251 versus predicted_max_residual=0.0005065036; confirmed=false.

Benchmark result