Hutch++ Curvature Controller
Source paper: Stochastic trace estimation for positive trace-class operators arXiv:2608.28135 ⓘ · analyzed Sep 2, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Replace the noisy Hutchinson estimate of a neural-network Hessian trace with a variance-reduced Hutch++ estimate computed only from Hessian-vector products. Use the estimated normalized curvature to cap or rescale the optimizer step, so learning-rate reductions occur when the loss landscape becomes globally sharp rather than when an individual minibatch gradient happens to be large.
Formulas
Mathematical statement
For a positive-semidefinite operator A on a Hilbert space, Hutch++ samples a Gaussian sketch S=[s_1,...,s_r], forms Y=AS, and lets Q have orthonormal columns spanning Y. It decomposes the trace into the exactly computed low-rank contribution tr(Q^T A Q) and the residual contribution of B=(I-QQ^T)A(I-QQ^T). With independent standard Gaussian probes g_j, the estimator is tr(Q^T A Q)+(1/m) sum_j g_j^T B g_j. Here A is a damped Hessian A=H+lambda I, H is the Hessian of the current training loss with respect to parameters, lambda>0 guarantees positive semidefiniteness, S and g_j are parameter-space Gaussian vectors, and Q^TQ=I. Hutchinson is unbiased because E[g^TBg]=tr(B); the projection removes dominant eigendirections before stochastic estimation, which is the variance-reduction mechanism. For a PSD residual, Hutch++ achieves the paper's stated improved O(epsilon^{-1}) rather than O(epsilon^{-2}) sample-product scaling for fixed relative accuracy, up to confidence and spectral-scale factors. The normalized curvature used by the controller is kappa_hat=tr_hat(A)/(d(1+lambda)), where d is the number of trainable parameters.
Implementation notes
Integrate this into the optimizer's training-time control path, not into the forward network. Every K optimizer steps, estimate the trace of the damped parameter Hessian A=H+lambda I using Hessian-vector products; obtain an HVP with Pearlmutter's method as grad_theta(dot(grad_theta(loss, theta, create_graph=True), v), theta), using the same fixed minibatch for all probes in one estimate. Use r=4 sketch vectors and m=8 residual vectors for the first MVP. Pseudocode: sample Gaussian S and g_j in parameter space; compute Y[:,i]=A(S[:,i]); QR-factor Y to obtain Q; compute T=Q^T A Q by applying A to every Q column; for each j compute u_j=g_j-Q(Q^Tg_j), apply A to u_j, and accumulate u_j^T A u_j; set trace_hat=trace(T)+mean(residuals); update eta=min(eta_max,c/(trace_hat/(d*(1+lambda))+eps)); then run AdamW or SGD with this eta. Estimate the actual trace from the paper's low-rank-plus-residual formula; estimate only the controller constant c, damping lambda, refresh period K, and confidence smoothing empirically. Start with a 2-layer MLP on CIFAR-10 or a small Transformer on WikiText-2, comparing AdamW with cosine decay and ordinary Hutchinson control at equal HVP count and equal optimizer steps. Log trace estimates, gradient norm, update norm, loss, and number of HVPs. Pre-register these mechanism tests: (1) at equal HVP budget, Hutch++ trace relative error versus a high-budget reference must be at least 2x lower than Hutchinson; (2) learning-rate drops should occur within one refresh interval after the estimated curvature crosses the controller boundary c/eta, with a correlation above 0.7; (3) removing the Q projection while preserving all probe counts should increase trace variance by at least 25% and produce more loss spikes. Falsify the transfer if Hutch++ has no lower variance than Hutchinson at equal HVP cost, or if curvature-controlled training does not improve the loss-versus-HVP curve and does not reduce instability.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.