Loss Landscape Features That Make Adam Stall: Definitions, Estimators, and the Preconditioned Hessian View

What the math gives to ML

The paper's transferable contribution is a matrix-free diagnosis of when Adam's diagonal scaling fails: the relevant object is the Hessian after Adam preconditioning, not the raw Hessian condition number. Its key distinction is between axis-aligned curvature, which diagonal scaling can fix, and cross-coupled curvature, which remains ill-conditioned in the preconditioned coordinates. A practical neural-network adaptation is a curvature-triggered hybrid optimizer that uses a few Hessian-vector products and Lanczos directions to add low-rank Newton corrections only when gradient energy lies in flat or badly conditioned preconditioned curvature bands. This preserves Adam's cheap default behavior while targeting the specific stall mechanism identified by the paper.