TD-to-PDE Continuation Training

Integrate this at the critic-training step of an actor-critic agent, or at the loss level of a neural operator or world model that has known drift and diffusion. Use an MLP or transformer critic $V_\theta(x)$ and obtain first- and second-order derivatives with automatic differentiation. For each minibatch, sample environment transitions for the TD term and independently sample collocation states from replay-buffer states, uniform domain points, or boundary-focused points for the PDE and boundary terms. Pseudocode: compute $\delta=r+\gamma V_\theta(x')-V_\theta(x)$ and $L_{TD}=\operatorname{mean}(\delta^2)$; evaluate $\nabla V_\theta$ and the Hessian at collocation states; form $\mathcal{L}^uV_\theta$; maximize over a discretized action set or evaluate at the current actor action $u=\pi_\phi(x)$; compute $L=L_{TD}+\lambda_{PDE}L_{PDE}+\lambda_BL_B$; update $\theta$; and update the actor separately using $\nabla_\phi V_\theta(x)$. Increase $\lambda_{PDE}$ by factor $q$ only when an exponential moving average of TD loss is below $\tau_{TD}$ and critic-output variance is below $\tau_V$. The SDE generator and PDE form are supplied by the known model; TD loss, output variance, residual estimates, and boundary error are measured empirically. First test on a 1D or 2D stochastic navigation task with an analytic or finite-difference reach-avoid solution. Compare pure PDE-PINN, pure TD critic, simultaneous fixed-weight training, and continuation training using identical networks and collocation points. The quantitative prediction is a sharp reduction in TD loss before strong PDE-residual reduction: staged training should avoid the early-loss plateau or divergence caused by a large fixed $\lambda_{PDE}$. Sweep the initial PDE weight and measure the largest stable value; continuation should tolerate a larger final $\lambda_{PDE}$ and achieve boundary error at least as low as fixed-weight training without increasing TD error by more than 20%.