Learning to Control Coupled-Dynamics Environments with Joint Markov Decision Processes

What the math gives to ML

The paper introduces Joint Markov Decision Processes in which one environment realization exposes coupled counterfactual outcomes for multiple actions, preserving dependencies that ordinary MDPs marginalize away. Its transferable asset is a distributional Bellman optimality operator over joint return laws, together with Wasserstein convergence guarantees and sampled neural targets. A promising implementation is a joint distributional critic that outputs returns for all actions under shared latent particles, rather than independently estimating one return distribution per action. This can improve counterfactual credit assignment and uncertainty estimation when an environment or simulator can provide common-randomness coupled transitions.