Robust Decentralized Optimization under Node Failures via Adaptive Regularization

What the math gives to ML

The paper introduces a constructive way to handle permanent data-source failures: replace a departing node's objective by a gradient-matched quadratic legacy rather than deleting its contribution. The transferable asset is a compact state consisting of an anchor, gradient, and curvature, together with an invariant-preserving correction that lets decentralized gradient tracking continue optimizing an augmented objective. This is directly applicable to federated or decentralized training with client churn, stragglers, device eviction, or privacy-driven client departure. The most promising first test is to maintain departed clients as cheap quadratic pseudo-losses and compare forgetting, communication, and convergence against simply dropping their data.