TOBYQA: A Trust-Region Method for Derivative-Free Optimization on Time-Varying Functions

What the math gives to ML

The paper develops a derivative-free trust-region surrogate that remains well posed when function evaluations drift over time and contain noise. Its transferable asset is the joint quadratic-plus-linear-time model, together with ridge regularization that replaces exact interpolation by a controlled fit and relaxes geometric poisedness requirements. This is especially useful for expensive neural-network hyperparameter or fine-tuning evaluations performed on rolling validation windows, where the objective changes between queries and repeated evaluations are impractical. The most direct adaptation is a drift-aware trust-region optimizer for hyperparameters, training schedules, or low-dimensional adapter configurations.