Variance-Reduced Q-Learning over Static and Time-Varying Networks

What the math gives to ML

The paper provides a constructive way to obtain nearly linear statistical speedups in decentralized Q-learning while using only a small number of communication rounds per learning epoch. Its transferable mathematical asset is the separation between Bellman contraction error and consensus error: repeated application of a doubly stochastic mixing matrix reduces disagreement geometrically as rho^L, allowing communication depth to be selected from an explicit error budget. The most promising neural-network adaptation is periodic consensus of distributed Q-network updates or TD-target statistics, combined with epoch-local aggregation, rather than communicating model parameters after every minibatch. This should be tested first in multi-agent or federated deep RL, where local data are independent but the MDP and optimal Q-function are shared.