Mechanism failed
2026
Spectral-filtered task-gradient optimizer
Replace the ordinary average of task or client gradients with an iterative spectral filter that removes tasks whose gradient vectors explain an anomalously large covariance direction. The global model uses the filtered gradient, while each task still maintains its own personalized parameters and local optimizer state. Unlike parameter-center regularization, the robustification acts directly on the vector messages and is designed to avoid an additional \(\sqrt d\) contamination penalty.
Paper: Contaminated Multi-task Learning with Heterogeneity: Fundamental Limits and Optimal Algorithms
arXiv:2607.02681