Gradient descent with exponentially increasing stepsizes and restarts

What the math gives to ML

The paper turns an apparently unstable exponentially increasing learning-rate schedule into a usable method by restarting exactly when consecutive update norms begin growing faster than the prescribed exponential schedule. The transferable asset is a derivative-free instability detector: it does not require Hessian estimates, gradient-noise modeling, or a well-calibrated initial learning rate, and it exploits the fact that an increasing step size searches for a favorable scale before diverging. For neural networks, the most direct experiment is a centralized optimizer wrapper around minibatch SGD, with global restarts that reset the learning-rate clock while preserving the parameters.