# Hutch++ Curvature Controller

- ID: 2933
- Canonical URL: https://synthcore.org/idea/2933/hutch-curvature-controller
- API JSON: https://synthcore.org/api/idea/2933.json
- API Markdown: https://synthcore.org/api/idea/2933.md
- Verification status: unverified
- Source: [arXiv:2608.28135](https://arxiv.org/abs/2608.28135)
- Category: optimization
- Solves: speedup, stability
- ML areas: optimizer, training-dynamics
- Math tags: linear-algebra, random-matrix, optimization
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace the noisy Hutchinson estimate of a neural-network Hessian trace with a variance-reduced Hutch++ estimate computed only from Hessian-vector products. Use the estimated normalized curvature to cap or rescale the optimizer step, so learning-rate reductions occur when the loss landscape becomes globally sharp rather than when an individual minibatch gradient happens to be large.

## Mathematical statement

For a positive-semidefinite operator A on a Hilbert space, Hutch++ samples a Gaussian sketch S=[s_1,...,s_r], forms Y=AS, and lets Q have orthonormal columns spanning Y. It decomposes the trace into the exactly computed low-rank contribution tr(Q^T A Q) and the residual contribution of B=(I-QQ^T)A(I-QQ^T). With independent standard Gaussian probes g_j, the estimator is tr(Q^T A Q)+(1/m) sum_j g_j^T B g_j. Here A is a damped Hessian A=H+lambda I, H is the Hessian of the current training loss with respect to parameters, lambda>0 guarantees positive semidefiniteness, S and g_j are parameter-space Gaussian vectors, and Q^TQ=I. Hutchinson is unbiased because E[g^TBg]=tr(B); the projection removes dominant eigendirections before stochastic estimation, which is the variance-reduction mechanism. For a PSD residual, Hutch++ achieves the paper's stated improved O(epsilon^{-1}) rather than O(epsilon^{-2}) sample-product scaling for fixed relative accuracy, up to confidence and spectral-scale factors. The normalized curvature used by the controller is kappa_hat=tr_hat(A)/(d(1+lambda)), where d is the number of trainable parameters.

## Key formulas

- $$Y=AS,\qquad Q=\operatorname{orth}(Y),\qquad B=(I-QQ^{T})A(I-QQ^{T})$$
- $$\widehat{\operatorname{tr}}_{\mathrm{Hutch++}}(A)=\operatorname{tr}(Q^{T}AQ)+\frac{1}{m}\sum_{j=1}^{m}g_j^{T}(I-QQ^{T})A(I-QQ^{T})g_j$$
- $$\widehat{\kappa}=\frac{\widehat{\operatorname{tr}}_{\mathrm{Hutch++}}(H+\lambda I)}{d(1+\lambda)},\qquad \eta_t=\min\!\left(\eta_{\max},\frac{c}{\widehat{\kappa}_t+\epsilon}\right)$$
- $$\mathbb{E}[g^{T}Bg]=\operatorname{tr}(B),\qquad B\succeq0$$

## Implementation notes

Integrate this into the optimizer's training-time control path, not into the forward network. Every K optimizer steps, estimate the trace of the damped parameter Hessian A=H+lambda I using Hessian-vector products; obtain an HVP with Pearlmutter's method as grad_theta(dot(grad_theta(loss, theta, create_graph=True), v), theta), using the same fixed minibatch for all probes in one estimate. Use r=4 sketch vectors and m=8 residual vectors for the first MVP. Pseudocode: sample Gaussian S and g_j in parameter space; compute Y[:,i]=A(S[:,i]); QR-factor Y to obtain Q; compute T=Q^T A Q by applying A to every Q column; for each j compute u_j=g_j-Q(Q^Tg_j), apply A to u_j, and accumulate u_j^T A u_j; set trace_hat=trace(T)+mean(residuals); update eta=min(eta_max,c/(trace_hat/(d*(1+lambda))+eps)); then run AdamW or SGD with this eta. Estimate the actual trace from the paper's low-rank-plus-residual formula; estimate only the controller constant c, damping lambda, refresh period K, and confidence smoothing empirically. Start with a 2-layer MLP on CIFAR-10 or a small Transformer on WikiText-2, comparing AdamW with cosine decay and ordinary Hutchinson control at equal HVP count and equal optimizer steps. Log trace estimates, gradient norm, update norm, loss, and number of HVPs. Pre-register these mechanism tests: (1) at equal HVP budget, Hutch++ trace relative error versus a high-budget reference must be at least 2x lower than Hutchinson; (2) learning-rate drops should occur within one refresh interval after the estimated curvature crosses the controller boundary c/eta, with a correlation above 0.7; (3) removing the Q projection while preserving all probe counts should increase trace variance by at least 25% and produce more loss spikes. Falsify the transfer if Hutch++ has no lower variance than Hutchinson at equal HVP cost, or if curvature-controlled training does not improve the loss-versus-HVP curve and does not reduce instability.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.