On the Universality of Simple Trust-Region Algorithms

What the math gives to ML

The paper identifies a useful optimizer principle: ordinary quadratic trust-region methods can adapt automatically to unknown Hessian regularity, without being given the Hölder exponent or its constant. The transferable asset is the coupling of model decrease, actual function decrease, and radius adaptation, allowing one method to behave differently in nonconvex, convex, and locally Newton-like regimes. A neural-network implementation should use Hessian-vector products or a cheap Gauss-Newton curvature operator, solve the trust-region subproblem approximately, and retain the acceptance-ratio controller. The main test is against AdamW and damped Newton at matched gradient and curvature-product budgets.