Disciplined Bilevel Programming

What the math gives to ML

The paper contributes an executable way to convert a convex lower-level optimization problem into a single nonlinear program by combining conic canonicalization, KKT conditions, and complementarity-gap continuation. This is transferable to constrained meta-learning and hyperparameter optimization, where the inner problem can be a convex adaptation, calibration, pruning, or sparse-estimation layer and the outer objective evaluates validation performance. The useful asset is not merely bilevel differentiation, which is already common, but a solver-oriented treatment of inequality constraints that avoids enforcing degenerate complementarity exactly from the first iteration. A practical neural-network adaptation is to solve the inner constrained module through a sequence of relaxed KKT problems with decreasing complementarity tolerance, then differentiate through the resulting approximate solution.