Recursive Bellman Variance Targets

Integration point: add a second output head to a value network, producing a mean v_theta(s,a) and nonnegative conditional variance q_theta(s,a)=softplus(h_theta(s,a))+epsilon. For each training transition or imagined branch, retain the child predictions instead of forming only a scalar return. Pseudocode: (1) obtain child means m_j and variances s_j^2; (2) compute z_j=r_j+m_j and m_rec=sum(p_j*z_j); (3) compute q_rec=sum(p_j*(s_j^2+u_j^2+2*c_j+(z_j-m_rec)^2)); (4) detach q_rec; (5) minimize 0.5*((G-m_theta)^2/(q_rec+eps)+log(q_rec+eps)), optionally with a separate stop-gradient regression target for q_theta. When only sampled branches are available, use node-local running counts, reward means, reward second moments, child means, and child variances; do not calculate variance over all complete paths as if they were independent. The paper mechanism supplies the recursive decomposition and estimator structure; transition probabilities, reward variances, and child statistics are estimated from replay or rollout batches. First cheap experiment: train a distributional/value network on CartPole or a small stochastic GridWorld, comparing ordinary MSE plus naive path variance against recursive heteroscedastic Bellman training. Use 1, 2, 4, 8, and 16 imagined branches. The quantitative prediction is that naive root-return variance underestimates held-out return squared error increasingly with branch depth, while recursive variance has calibration ratio E[(G-v)^2]/E[q_rec] near 1.0 and remains within 20% across depths. A second predicted signature is monotonic variance growth with horizon under nonzero transition noise; removing transition randomness should eliminate the between-child term and sharply reduce the gap between recursive and naive estimates.