SGHA: A Single-Loop Fully First-Order Algorithm for Nonconvex-Strongly-Convex Bilevel Optimization

What the math gives to ML

The paper gives a constructive way to remove the inner optimization loop in nonconvex–strongly-convex bilevel problems: lower-level stationarity is treated as an equality constraint and solved with a bounded, regularized primal-dual method. Its transferable asset is the use of finite differences of ordinary gradients to estimate mixed Hessian-vector products, avoiding explicit second-order autodiff. The most promising neural-network application is single-loop hyperparameter optimization or meta-learning, where outer and inner parameters are updated simultaneously rather than through repeated inner optimization.