# Value-Gradient Trajectory Collocation

- ID: 2130
- Canonical URL: https://synthcore.org/idea/2130/value-gradient-trajectory-collocation
- API JSON: https://synthcore.org/api/idea/2130.json
- API Markdown: https://synthcore.org/api/idea/2130.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.11480](https://arxiv.org/abs/2608.11480)
- Category: training
- Solves: accuracy, sample-efficiency, stability
- ML areas: training-dynamics, optimizer, world-model, rl
- Math tags: control-theory, dynamical-systems, pde, stochastic-processes, optimization
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace a static or uniformly random PINN collocation distribution with points generated by rolling out the model's own local feedback dynamics. For a learned scalar field V_theta(x,t), compute a control and adversarial direction from grad_x V_theta, integrate the physical dynamics forward, add controlled Gaussian exploration, and train on the resulting points together with a small uniform reservoir. This should concentrate samples near reachable boundaries, large-residual regions, and dynamically important trajectories while retaining coverage of the global domain.

## Mathematical statement

For dynamics dot{x}=f(x,u,d), let V_theta(x,t) be the neural value function and let p=grad_x V_theta(x,t). Define the Hamiltonian H(x,p)=min_{u in U} max_{d in D} p^T f(x,u,d), with the min/max reversed if the application uses the opposite value convention. At each rollout state, choose approximate actions (u*,d*) by solving these bounded inner optimizations, then evolve a stochastic steering process dX_s=f(X_s,u*(X_s),d*(X_s))ds+sqrt(2 sigma^2)dW_s. Collocation samples are drawn from the empirical occupation measure rho_hat(x)=(NK)^(-1) sum_k sum_j delta(x-X_kj), optionally mixed with uniform density rho_train=(1-alpha)rho_uniform+alpha rho_hat. The PINN minimizes the HJ residual R_theta=partial_t V_theta+H(x,grad_x V_theta) and boundary or terminal residuals. The transferable mechanism is feedback-induced concentration, while sigma greater than zero prevents collapse onto a few deterministic characteristics.

## Key formulas

- $$H(x,p)=\min_{u\in\mathcal U}\max_{d\in\mathcal D}p^\top f(x,u,d),\qquad p=\nabla_xV_\theta(x,t),$$
- $$R_\theta(x,t)=\partial_tV_\theta(x,t)+H\!\left(x,\nabla_xV_\theta(x,t)\right),$$
- $$dX_s=f\!\left(X_s,u^*(X_s),d^*(X_s)\right)ds+\sqrt{2\sigma^2}\,dW_s,$$
- $$\rho_{\mathrm{train}}=(1-\alpha)\rho_{\mathrm{uniform}}+\alpha\hat\rho_{\mathrm{traj}},\qquad \hat\rho_{\mathrm{traj}}=\frac{1}{NK}\sum_{k=1}^{N}\sum_{j=1}^{K}\delta_{X_{k,j}}.$$

## Implementation notes

1. Exact integration point: use this sampler between optimizer updates for any PINN, neural ODE residual model, or value/world-model network. Maintain a replay buffer containing a fixed fraction of uniform domain points and a steerable buffer containing states from current forward rollouts. At a sampled (x,t), evaluate V_theta and its input gradient with autodiff. For box-bounded controls and disturbances, first implement the cheap affine case: calculate g_u=(grad_x V_theta)^T f_u and g_d=(grad_x V_theta)^T f_d, then choose each component of u* and d* at the corresponding bound according to the sign of its coefficient; for nonlinear dynamics use 3-5 projected-gradient steps or a small random candidate set. 2. Pseudocode: initialize uniform points; repeat: compute p=grad_x(V_theta(x,t)); solve u_star=argmin_u p^T f(x,u,d) and d_star=argmax_d p^T f(x,u,d); set x_next=x+Delta_t f(x,u_star,d_star)+sqrt(2 sigma^2 Delta_t) epsilon, where epsilon is standard Gaussian noise; append x_next and intermediate states to the trajectory buffer; draw a batch with alpha trajectory points and 1-alpha uniform points; minimize mean(R_theta^2)+lambda_B mean(boundary_residual^2); periodically refresh the buffer because the policy induced by V_theta changes. Clip states to the PDE domain and reject or reflect invalid points. 3. Computed from the paper's mechanism: the feedback direction from the current value gradient, forward trajectory steering, and injected exploration. Estimated empirically: the occupation density and suitable alpha, Delta_t, and sigma. Use detached gradients for sampling so the sampler does not add second-order terms to the training loss. 4. First cheap experiment: solve a 2-D Burgers or linear-advection PINN and a 2-D double-integrator reachability HJ equation. Compare uniform Sobol sampling, residual-based adaptive sampling, and this method at equal network evaluations. Use a 4-layer tanh MLP, 10,000 collocation points, alpha=0.7, and noise levels sigma in {0, 0.01, 0.05}. The quantitative prediction is that the steered sampler lowers mean squared residual in the top 10 percent largest-error states by at least 25 percent at equal sample count and produces a sharper concentration of points within a prescribed distance of the learned zero level set. Sweep the control gain or steering strength: excessive steering should cause mode collapse and worse global error, while uniform mixing should prevent it. Measure the collapse transition by the effective sample size of occupied spatial bins; the useful regime should have high boundary coverage and lower residual, whereas sigma=0 should show visibly lower coverage and greater run-to-run variance.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: no_effect
- Verdict: Built Value-Gradient Trajectory Collocation with affine bounded min/max actions, stochastic steering, uniform-reservoir mixing, and a small PINN experiment. The checks confirmed zero-error affine corner selection, deterministic displacement scaling as gamma·dt, Brownian variance scaling as 2·sigma²·dt, and exact trajectory mixture fractions. The trajectory sampler did not improve this toy PINN: global residual MSE was 0.000706 versus 0.000359 for uniform sampling, and top-error MSE was 0.00246 versus 0.00116, so the mechanism was validated but the promised accuracy win was not observed.

### Mechanism check

- Verdict: Built Value-Gradient Trajectory Collocation with affine bounded min/max actions, stochastic steering, uniform-reservoir mixing, and a small PINN experiment. The checks confirmed zero-error affine corner selection, deterministic displacement scaling as gamma·dt, Brownian variance scaling as 2·sigma²·dt, and exact trajectory mixture fractions. The trajectory sampler did not improve this toy PINN: global residual MSE was 0.000706 versus 0.000359 for uniform sampling, and top-error MSE was 0.00246 versus 0.00116, so the mechanism was validated but the promised accuracy win was not observed.
- Confidence: 6/10
- Limitations: Only a small 1D toy steering field and 1D Hamilton-Jacobi PINN were tested; no 2D Burgers, double-integrator reachability, Sobol baseline, residual-based adaptive baseline, FLOP-normalized study, or extensive alpha/sigma/gain sweeps were run.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.4855
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/857/bench_report.json)
- [experiment.py](https://synthcore.org/code/857/experiment.py)
- [report.md](https://synthcore.org/code/857/report.md)
- [report_bench_2026-09-01T131251.md](https://synthcore.org/code/857/report_bench_2026-09-01T131251.md)
- [results.json](https://synthcore.org/code/857/results.json)
- [stage2_bench.py](https://synthcore.org/code/857/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/857)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.