{
 "artifacts": null,
 "category": "optimization",
 "description": "Replace the noisy Hutchinson estimate of a neural-network Hessian trace with a variance-reduced Hutch++ estimate computed only from Hessian-vector products. Use the estimated normalized curvature to cap or rescale the optimizer step, so learning-rate reductions occur when the loss landscape becomes globally sharp rather than when an individual minibatch gradient happens to be large.",
 "formulas_latex": [
  "$$Y=AS,\\qquad Q=\\operatorname{orth}(Y),\\qquad B=(I-QQ^{T})A(I-QQ^{T})$$",
  "$$\\widehat{\\operatorname{tr}}_{\\mathrm{Hutch++}}(A)=\\operatorname{tr}(Q^{T}AQ)+\\frac{1}{m}\\sum_{j=1}^{m}g_j^{T}(I-QQ^{T})A(I-QQ^{T})g_j$$",
  "$$\\widehat{\\kappa}=\\frac{\\widehat{\\operatorname{tr}}_{\\mathrm{Hutch++}}(H+\\lambda I)}{d(1+\\lambda)},\\qquad \\eta_t=\\min\\!\\left(\\eta_{\\max},\\frac{c}{\\widehat{\\kappa}_t+\\epsilon}\\right)$$",
  "$$\\mathbb{E}[g^{T}Bg]=\\operatorname{tr}(B),\\qquad B\\succeq0$$"
 ],
 "id": 2933,
 "implementation": "Integrate this into the optimizer's training-time control path, not into the forward network. Every K optimizer steps, estimate the trace of the damped parameter Hessian A=H+lambda I using Hessian-vector products; obtain an HVP with Pearlmutter's method as grad_theta(dot(grad_theta(loss, theta, create_graph=True), v), theta), using the same fixed minibatch for all probes in one estimate. Use r=4 sketch vectors and m=8 residual vectors for the first MVP. Pseudocode: sample Gaussian S and g_j in parameter space; compute Y[:,i]=A(S[:,i]); QR-factor Y to obtain Q; compute T=Q^T A Q by applying A to every Q column; for each j compute u_j=g_j-Q(Q^Tg_j), apply A to u_j, and accumulate u_j^T A u_j; set trace_hat=trace(T)+mean(residuals); update eta=min(eta_max,c/(trace_hat/(d*(1+lambda))+eps)); then run AdamW or SGD with this eta. Estimate the actual trace from the paper's low-rank-plus-residual formula; estimate only the controller constant c, damping lambda, refresh period K, and confidence smoothing empirically. Start with a 2-layer MLP on CIFAR-10 or a small Transformer on WikiText-2, comparing AdamW with cosine decay and ordinary Hutchinson control at equal HVP count and equal optimizer steps. Log trace estimates, gradient norm, update norm, loss, and number of HVPs. Pre-register these mechanism tests: (1) at equal HVP budget, Hutch++ trace relative error versus a high-budget reference must be at least 2x lower than Hutchinson; (2) learning-rate drops should occur within one refresh interval after the estimated curvature crosses the controller boundary c/eta, with a correlation above 0.7; (3) removing the Q projection while preserving all probe counts should increase trace variance by at least 25% and produce more loss spikes. Falsify the transfer if Hutch++ has no lower variance than Hutchinson at equal HVP cost, or if curvature-controlled training does not improve the loss-versus-HVP curve and does not reduce instability.",
 "math_summary": "For a positive-semidefinite operator A on a Hilbert space, Hutch++ samples a Gaussian sketch S=[s_1,...,s_r], forms Y=AS, and lets Q have orthonormal columns spanning Y. It decomposes the trace into the exactly computed low-rank contribution tr(Q^T A Q) and the residual contribution of B=(I-QQ^T)A(I-QQ^T). With independent standard Gaussian probes g_j, the estimator is tr(Q^T A Q)+(1/m) sum_j g_j^T B g_j. Here A is a damped Hessian A=H+lambda I, H is the Hessian of the current training loss with respect to parameters, lambda\u003e0 guarantees positive semidefiniteness, S and g_j are parameter-space Gaussian vectors, and Q^TQ=I. Hutchinson is unbiased because E[g^TBg]=tr(B); the projection removes dominant eigendirections before stochastic estimation, which is the variance-reduction mechanism. For a PSD residual, Hutch++ achieves the paper's stated improved O(epsilon^{-1}) rather than O(epsilon^{-2}) sample-product scaling for fixed relative accuracy, up to confidence and spectral-scale factors. The normalized curvature used by the controller is kappa_hat=tr_hat(A)/(d(1+lambda)), where d is the number of trainable parameters.",
 "math_tags": [
  "linear-algebra",
  "random-matrix",
  "optimization"
 ],
 "ml_areas": [
  "optimizer",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2608.28135",
  "arxiv_url": "https://arxiv.org/abs/2608.28135",
  "summary_what_math_gives_to_ml": "The paper develops matrix-free trace estimators for positive trace-class operators, extending Hutchinson and Hutch++ to infinite-dimensional settings. The transferable asset is variance reduction from projecting out a low-rank subspace discovered through operator-vector products, reducing the number of expensive products needed for reliable trace estimation from roughly O(epsilon^-2) to O(epsilon^-1) in the positive-semidefinite case. A neural-network use is to estimate the trace of a damped Hessian or empirical Fisher through Hessian-vector products, enabling curvature-controlled learning-rate schedules or regularization without materializing the matrix. The most defensible first transfer is a Hutch++ curvature monitor whose predictions can be tested against ordinary Hutchinson at equal Hessian-vector-product cost.",
  "title": "Stochastic trace estimation for positive trace-class operators",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 5,
  "novelty": 6,
  "usefulness": 6
 },
 "solves": [
  "speedup",
  "stability"
 ],
 "title": "Hutch++ Curvature Controller",
 "url": "https://synthcore.org/idea/2933/hutch-curvature-controller",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}