# Gradient-Adaptive Parameter-Free Cubic Newton

- ID: 703
- Canonical URL: https://synthcore.org/idea/703/gradient-adaptive-parameter-free-cubic-newton
- API JSON: https://synthcore.org/api/idea/703.json
- API Markdown: https://synthcore.org/api/idea/703.md
- Verification status: unverified
- Source: [arXiv:2607.10741](https://arxiv.org/abs/2607.10741)
- Category: optimization
- Solves: speedup, stability, accuracy
- ML areas: optimizer, training-dynamics
- Math tags: optimization, numerical-analysis, convex-analysis
- Ratings: usefulness 6/10; difficulty 7/10; novelty 6/10

## Idea description

Replace a fixed-cubic-regularized Newton step with an adaptive cubic model whose coefficient is increased when the observed loss violates the local Taylor model. The regularizer becomes stronger automatically in regions with large gradients, reflecting the paper's generalized smoothness law, while shrinking near stationary points so that Newton curvature is used more aggressively.

## Mathematical statement

The paper assumes the generalized third-order smoothness condition ||∇^3f(x)|| ≤ L0 + L1||∇f(x)||, where f:R^d→R is three-times continuously differentiable, ∇^3f(x) is the operator norm of the third derivative tensor, and L0,L1≥0 are unknown problem constants. It targets an (ε,δ)-second-order stationary point satisfying ||∇f(x)||≤ε and ∇^2f(x)≽−δI, with I the identity matrix. The adapted neural optimizer uses the cubic model m_k(s)=g_k^T s + 1/2 s^T H_k s + σ_k||s||_2^3/6, where g_k=∇f(x_k), H_k=∇^2f(x_k), s is a parameter update, and σ_k>0 is an unknown local regularization coefficient. Instead of knowing L0 and L1, it backtracks σ_k until the actual decrease satisfies the cubic Taylor acceptance inequality f(x_k+s_k)≤f(x_k)+g_k^T s_k+1/2 s_k^T H_k s_k+σ_k||s_k||_2^3/6. The gradient-aware initialization σ_k=σ_min+c||g_k||_2 is an operational surrogate for L0+L1||∇f(x_k)||, with c≥0 estimated or selected once.

## Key formulas

- $$\|\nabla^{3}f(x)\|\leq L_{0}+L_{1}\|\nabla f(x)\|.$$
- $$\|\nabla f(x_k)\|\leq\varepsilon,\qquad \nabla^{2}f(x_k)\succeq-\delta I.$$
- $$m_k(s)=g_k^{\mathsf T}s+\frac12s^{\mathsf T}H_ks+\frac{\sigma_k}{6}\|s\|_2^3,$$
- $$f(x_k+s_k)\leq f(x_k)+g_k^{\mathsf T}s_k+\frac12s_k^{\mathsf T}H_ks_k+\frac{\sigma_k}{6}\|s_k\|_2^3.$$

## Implementation notes

Integrate this at the optimizer-update level, initially for a small trainable model or one parameter block rather than an entire billion-parameter network. At iteration k, compute the gradient g and obtain Hessian-vector products H v through automatic differentiation; approximately minimize the cubic model with 5–20 Lanczos or conjugate-gradient iterations. Use this pseudocode: (1) set sigma = sigma_min + c*norm(g), with sigma_min=1e-4 and c initially 0.1; (2) approximately solve s = argmin_s g^T s + 0.5*s^T H*s + sigma*norm(s)^3/6 using cubic-CG or repeated Newton solves; (3) clip norm(s) to a prescribed trust radius if necessary; (4) evaluate the loss on a held-out microbatch at x+s; (5) accept if f_trial <= f_current + g^T s + 0.5*s^T H*s + sigma*norm(s)^3/6 and the actual reduction is positive; otherwise set sigma <- 2*sigma and recompute; (6) after two consecutive strong acceptances, set sigma <- max(sigma/2,sigma_min). Apply x <- x+s only after acceptance. The mathematical quantities supplied by the method are the generalized-smoothness surrogate and cubic Taylor condition; the implementation estimates Hessian products, the cubic subproblem solution, and acceptance behavior empirically. Use full-batch or 4–8x larger validation microbatches for acceptance because ordinary minibatch noise can falsely reject steps. First test on a 2–4 layer MLP or small ResNet on CIFAR-10, comparing against AdamW, Newton-CG with a fixed cubic coefficient, and a trust-region baseline at equal Hessian-vector-product budgets. Measure loss decrease per HVP, rejected-step fraction, gradient norm, minimum Hessian eigenvalue estimated by Lanczos, and final accuracy. Success means at least 20% lower training loss at equal HVP budget, fewer divergence spikes, or improved accuracy at the same number of optimizer steps.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.