# TD-to-PDE Continuation Training

- ID: 2433
- Canonical URL: https://synthcore.org/idea/2433/td-to-pde-continuation-training
- API JSON: https://synthcore.org/api/idea/2433.json
- API Markdown: https://synthcore.org/api/idea/2433.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.17117](https://arxiv.org/abs/2608.17117)
- Category: training
- Solves: stability, accuracy, sample-efficiency
- ML areas: optimizer, rl, training-dynamics, world-model, regularization
- Math tags: control-theory, dynamical-systems, pde, stochastic-processes, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Train a neural value or latent-dynamics model with temporal-difference targets before enforcing a stiff differential-equation residual, and ramp the physics weight only after the critic has become predictive. For a stochastic dynamical model, the residual is computed using the infinitesimal generator, while terminal, safe, and failure boundary conditions are imposed through separate penalties.

## Mathematical statement

For the stochastic system $\mathrm{d}X_t=f(X_t,U_t)\mathrm{d}t+\sigma(X_t,U_t)\mathrm{d}W_t$, define $a(x,u)=\sigma(x,u)\sigma(x,u)^\top$ and the generator $\mathcal{L}^u V(x)=f(x,u)^\top\nabla V(x)+\frac{1}{2}\operatorname{tr}(a(x,u)\nabla^2V(x))$. A candidate reach-avoid value $V_\theta(x)$ should satisfy a stationary or discounted Bellman-HJB equation of the form $\mathcal{H}[V_\theta](x)=0$, where $\mathcal{H}[V](x)=\max_{u\in\mathbb{U}}\{\mathcal{L}^uV(x)-\beta V(x)+r(x,u)\}$ for control maximization, discount rate $\beta\geq0$, and reward/source term $r$. Given transitions $(x_t,u_t,x_{t+1},r_t)$, the TD error is $\delta_t=r_t+\gamma V_\theta(x_{t+1})-V_\theta(x_t)$, with discount factor $\gamma$. Optimize $L(\theta;\lambda)=L_{TD}(\theta)+\lambda_{PDE}L_{PDE}(\theta)+\lambda_B L_B(\theta)$, where $L_{PDE}=\mathbb{E}_{x\sim\mu}|\mathcal{H}[V_\theta](x)|^2$ and $L_B$ penalizes violations of prescribed boundary or terminal values $g(x)$. The mechanism is continuation in $\lambda_{PDE}$: begin at $\lambda_{PDE}=0$ and increase it only after TD error and value variance fall below specified thresholds.

## Key formulas

- $$\mathrm{d}X_t=f(X_t,U_t)\,\mathrm{d}t+\sigma(X_t,U_t)\,\mathrm{d}W_t,\qquad a(x,u)=\sigma(x,u)\sigma(x,u)^\top.$$
- $$\mathcal{L}^{u}V(x)=f(x,u)^\top\nabla V(x)+\frac{1}{2}\operatorname{tr}\!\left(a(x,u)\nabla^2V(x)\right),\qquad \mathcal{H}[V](x)=\max_{u\in\mathbb{U}}\{\mathcal{L}^{u}V(x)-\beta V(x)+r(x,u)\}.$$
- $$\delta_t=r_t+\gamma V_\theta(x_{t+1})-V_\theta(x_t),\qquad L(\theta)=\mathbb{E}[\delta_t^2]+\lambda_{PDE} \mathbb{E}_{x\sim\mu}\left[\mathcal{H}[V_\theta](x)^2\right]+\lambda_B\mathbb{E}_{x\sim\partial\Omega}\left[V_\theta(x)-g(x)\right]^2.$$
- $$\lambda_{PDE}^{(k+1)}=\min\!\left(\lambda_{\max},\,q\lambda_{PDE}^{(k)}\right)\quad\text{if}\quad \widehat{L}_{TD}<\tau_{TD}\ \text{and}\ \widehat{\operatorname{Var}}[V_\theta(X)]<\tau_V,\qquad q>1.$$

## Implementation notes

Integrate this at the critic-training step of an actor-critic agent, or at the loss level of a neural operator or world model that has known drift and diffusion. Use an MLP or transformer critic $V_\theta(x)$ and obtain first- and second-order derivatives with automatic differentiation. For each minibatch, sample environment transitions for the TD term and independently sample collocation states from replay-buffer states, uniform domain points, or boundary-focused points for the PDE and boundary terms. Pseudocode: compute $\delta=r+\gamma V_\theta(x')-V_\theta(x)$ and $L_{TD}=\operatorname{mean}(\delta^2)$; evaluate $\nabla V_\theta$ and the Hessian at collocation states; form $\mathcal{L}^uV_\theta$; maximize over a discretized action set or evaluate at the current actor action $u=\pi_\phi(x)$; compute $L=L_{TD}+\lambda_{PDE}L_{PDE}+\lambda_BL_B$; update $\theta$; and update the actor separately using $\nabla_\phi V_\theta(x)$. Increase $\lambda_{PDE}$ by factor $q$ only when an exponential moving average of TD loss is below $\tau_{TD}$ and critic-output variance is below $\tau_V$. The SDE generator and PDE form are supplied by the known model; TD loss, output variance, residual estimates, and boundary error are measured empirically. First test on a 1D or 2D stochastic navigation task with an analytic or finite-difference reach-avoid solution. Compare pure PDE-PINN, pure TD critic, simultaneous fixed-weight training, and continuation training using identical networks and collocation points. The quantitative prediction is a sharp reduction in TD loss before strong PDE-residual reduction: staged training should avoid the early-loss plateau or divergence caused by a large fixed $\lambda_{PDE}$. Sweep the initial PDE weight and measure the largest stable value; continuation should tolerate a larger final $\lambda_{PDE}$ and achieve boundary error at least as low as fixed-weight training without increasing TD error by more than 20%.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a 1-D OU diffusion critic experiment with automatic-differentiation generator residuals, TD loss, boundary penalty, fixed-PDE, TD-only, and gated continuation schedules. The generator check passed exactly, and weak-Euler error decreased from 0.00796 to 0.000922 as dt fell by 8x, close to the predicted linear ratio 0.125 (observed 0.116). The continuation gate showed a threshold transition, and activated continuation achieved lower PDE loss than fixed weighting, although fixed weighting had slightly better value RMSE and TD loss; this is a promising mechanism signal rather than a decisive overall win.

### Mechanism check

- Verdict: Built a 1-D OU diffusion critic experiment with automatic-differentiation generator residuals, TD loss, boundary penalty, fixed-PDE, TD-only, and gated continuation schedules. The generator check passed exactly, and weak-Euler error decreased from 0.00796 to 0.000922 as dt fell by 8x, close to the predicted linear ratio 0.125 (observed 0.116). The continuation gate showed a threshold transition, and activated continuation achieved lower PDE loss than fixed weighting, although fixed weighting had slightly better value RMSE and TD loss; this is a promising mechanism signal rather than a decisive overall win.
- Confidence: 8/10
- Limitations: Only a small manufactured 1-D diffusion and one random seed were tested; no actor learning, action maximization, reach-avoid boundaries, stochastic-policy evaluation, hyperparameter robustness, wall-clock comparison, or larger architectures were evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/943/bench_report.json)
- [poisson_track.py](https://synthcore.org/code/943/poisson_track.py)
- [report.md](https://synthcore.org/code/943/report.md)
- [report_bench_2026-09-01T162428.md](https://synthcore.org/code/943/report_bench_2026-09-01T162428.md)
- [results.json](https://synthcore.org/code/943/results.json)
- [run_bench.py](https://synthcore.org/code/943/run_bench.py)
- [run_experiment.py](https://synthcore.org/code/943/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/943)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.