A Decomposed Bilevel Search for Variable-Metric Proximal Gradient Methods

What the math gives to ML

The paper introduces a diagonal-plus-rank-one factorization for variable-metric proximal steps, replacing a difficult high-dimensional quasi-Newton proximal problem by repeated ordinary diagonal-metric proximal evaluations and a low-dimensional monotone root solve. This is directly transferable to neural-network training whenever the regularizer has a cheap diagonal proximal operator, such as group sparsity, sorted sparsity, or structured pruning penalties. The most practical adaptation is a proximal-gradient optimizer whose metric is a positive diagonal matrix plus one curvature-informed rank-one correction, with the rank-one scalar solved by safeguarded bisection and warm-started across iterations.