Parameter-Free Dynamic Regret for Online Convex Optimization under Heavy-Tailed Noise

What the math gives to ML

This paper provides a principled way to adapt online optimization to two conditions common in neural-network training but poorly handled by ordinary optimizers: drifting objectives and gradient noise with only finite moments below two. Its transferable asset is the combination of restarted AdaGrad experts at geometrically spaced time scales with a pathwise meta-algorithm that does not require knowing the noise level, comparator drift, horizon, or Lipschitz bound. The most direct neural-network use is a small ensemble of optimizer trajectories for a drifting fine-tuning problem, such as LoRA or the final layer, with adaptive online weighting selecting the appropriate time scale. The supplied extraction does not expose the paper's full AdaGrad-Hedge update, so the implementation below gives an explicit paper-inspired normalized Hedge rule and treats the stated regret bound as the design target rather than claiming its theorem applies unchanged to nonconvex networks.