Replace the ordinary average of task or client gradients with an iterative spectral filter that removes tasks whose gradient vectors explain an anomalously large covariance direction. The global model uses the filtered gradient, while each task still maintains its own personalized parameters and local optimizer state. Unlike parameter-center regularization, the robustification acts directly on the vector messages and is designed to avoid an additional \(\sqrt d\) contamination penalty.
Replace ordinary Wasserstein or arithmetic pooling of distribution-valued features with a barycenter whose individual quantile displacements are Huberized. Small changes between input distributions remain averaged quadratically, while a corrupted token, expert, graph neighborhood, or augmentation cannot move the pooled distribution arbitrarily far. The module is especially cheap for one-dimensional distributions represented by fixed quantile vectors.
For models that predict probability distributions, replace the usual Wasserstein-2 loss or a Huber penalty on the final Wasserstein distance with a Huber penalty on quantile-by-quantile prediction errors. This suppresses gradients from localized outliers while retaining quadratic gradients on the majority of the distribution, which is useful for uncertainty prediction, histogram prediction, and distributional distillation.