Consistent Variance Estimation for Q-Function Estimators in Finite-Horizon MDP Tree Search

What the math gives to ML

The paper offers a concrete law-of-total-variance decomposition for finite-horizon tree returns and shows that treating sampled root-to-leaf paths as i.i.d. produces an asymptotically persistent downward bias. Its transferable asset is a node-local recursive estimator that propagates uncertainty through branching transitions, rather than recomputing a misleading sample variance over correlated trajectories. The most promising neural-network transfer is uncertainty-aware Bellman training for world models, value networks, and branching rollout systems: propagate reward, transition, and child-value variance separately, then use the resulting variance to weight targets, prioritize data, or stop rollouts.