# Sensitivity-Particle Training for Marginal-Only Latent ODEs

- ID: 1878
- Canonical URL: https://synthcore.org/idea/1878/sensitivity-particle-training-for-marginal-only-latent-odes
- API JSON: https://synthcore.org/api/idea/1878.json
- API Markdown: https://synthcore.org/api/idea/1878.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.06190](https://arxiv.org/abs/2608.06190)
- Category: training
- Solves: accuracy, sample-efficiency, stability
- ML areas: world-model, training, optimizer
- Math tags: dynamical-systems, probability, statistics, numerical-analysis
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Train an augmented latent neural ODE from snapshot observations of only the visible coordinates by transporting particles from an initial latent distribution and differentiating their visible locations through forward sensitivity equations. Replace density-PDE discretization or potentially biased same-particle density objectives with a kernel marginal-matching loss whose gradient is estimated using independent particle sets.

## Mathematical statement

The paper starts from the generalized ODE \(\dot{q}(t)=R(q(t),t;\alpha)\) with \(q(0)=q_0\), where \(q=(x,y)\) contains visible coordinates \(x\in\mathbb{R}^{d_x}\) and hidden coordinates \(y\in\mathbb{R}^{d_y}\), and \(\alpha\) is the unknown parameter vector. Let \(\Phi_t(q_0;\alpha)\) be the ODE flow, \(Q_i(t)=\Phi_t(Q_i(0);\alpha)\) a simulated particle, and \(X_i(t)=\pi_xQ_i(t)\) its visible projection. The forward sensitivity is \(S_i(t)=\partial Q_i(t)/\partial\alpha\in\mathbb{R}^{d\times p}\) and satisfies \(\dot{S}_i=J_R(Q_i,t;\alpha)S_i+\partial_\alpha R(Q_i,t;\alpha)\), with \(S_i(0)=\partial_\alpha Q_i(0)\). For a kernel \(K_h(u)=h^{-d_x}K(u/h)\), the simulated marginal density is \(\hat p_\alpha(z,t)=N^{-1}\sum_iK_h(z-X_i(t))\), and \(\partial_\alpha K_h(z-X_i)=-\nabla_zK_h(z-X_i)\,\pi_xS_i\). A practical crossed estimator uses an observed minibatch \(Z\) and independent simulated particle sets \(A\) and \(B\). The objective is \(\mathcal{L}=\mathbb{E}[K_h(Z-X)]-\tfrac{1}{2}\mathbb{E}[K_h(X-X')]\), where \(X\) and \(X'\) are independent simulated visible particles. Its gradient estimator is \(\widehat{\nabla_\alpha\mathcal{L}}=(|Z||A|)^{-1}\sum_{z\in Z,i\in A}\nabla_\alpha K_h(z-X_i)-(|A||B|)^{-1}\sum_{i\in A,j\in B}\nabla_\alpha K_h(X_i-X_j)\).

## Key formulas

- $$\dot{q}(t)=R(q(t),t;\alpha),\qquad q(0)=q_0,\qquad q=(x,y)\in\mathbb{R}^{d_x+d_y}.$$
- $$\dot{S}(t)=J_R(q(t),t;\alpha)S(t)+\partial_\alpha R(q(t),t;\alpha),\qquad S(0)=\partial_\alpha q_0.$$
- $$\hat p_\alpha(z,t)=\frac{1}{N}\sum_{i=1}^N K_h(z-X_i(t)),\qquad \partial_\alpha K_h(z-X_i)=-\nabla_zK_h(z-X_i)\,\pi_xS_i(t).$$
- $$\widehat{\nabla_\alpha\mathcal{L}}=\frac{1}{|Z||A|}\sum_{z\in Z,i\in A}\nabla_\alpha K_h(z-X_i)-\frac{1}{|A||B|}\sum_{i\in A,j\in B}\nabla_\alpha K_h(X_i-X_j),\quad \mathcal{L}=\mathbb{E}K_h(Z-X)-\frac12\mathbb{E}K_h(X-X').$$

## Implementation notes

Integrate this at the training step of a latent neural ODE or continuous-time world model. Let the neural vector field be \(R_\theta(q,t)\), with \(q=[x,y]\) and a low-dimensional latent state \(y\). Initialize simulated particles by sampling visible initial states from the dataset and latent initial states from a learned diagonal Gaussian or an encoder. For each particle and each requested snapshot time, integrate the augmented state \((q_i,S_i)\) with an adaptive ODE solver. Obtain \(J_R S_i\) and \(\partial_\theta R\) using automatic-differentiation Jacobian-vector products rather than explicitly constructing a dense Jacobian. Split particles into independent sets A and B. For an observed minibatch Z, compute the Gaussian kernel \(K_h(u)=(2\pi h^2)^{-d_x/2}\exp(-\lVert u\rVert^2/(2h^2))\), evaluate the two crossed sums in the gradient formula, and update \(\theta\) with AdamW. Use bandwidth \(h\) proportional to the median pairwise distance in the observed minibatch, clipped to a fixed range; resample latent initial states each iteration. The paper supplies the characteristic sensitivity recursion and unbiased crossed-estimator structure; bandwidth, particle count, solver tolerance, and latent prior are empirical settings. The first cheap experiment should use a two-dimensional bistable ODE with only the first coordinate observed, 64 particles per set, and a small MLP vector field. Compare against a same-particle KDE objective and a deterministic mean-field neural ODE. Evaluate held-out marginal kernel distance, mode locations, trajectory stability, and gradient variance. Success is lower held-out marginal error and smoother loss descent at equal particle count, particularly when the observed marginal is bimodal or observations are temporally sparse.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a reproducible sensitivity-particle KDE experiment with exact forward sensitivities, crossed independent-particle gradients, same-particle comparison, and equal-budget training. Finite-difference sensitivity error scaled as eps^2 (observed slope 1.9998 vs 2), crossed gradient variance scaled approximately as N^-1 (observed slope -0.9008), and the same-minus-cross difference showed 1/N scaling. Crossed training obtained a higher held-out objective than the same-particle baseline (0.3746 vs 0.3658), although this is only a small toy result.

### Mechanism check

- Verdict: Built a reproducible sensitivity-particle KDE experiment with exact forward sensitivities, crossed independent-particle gradients, same-particle comparison, and equal-budget training. Finite-difference sensitivity error scaled as eps^2 (observed slope 1.9998 vs 2), crossed gradient variance scaled approximately as N^-1 (observed slope -0.9008), and the same-minus-cross difference showed 1/N scaling. Crossed training obtained a higher held-out objective than the same-particle baseline (0.3746 vs 0.3658), although this is only a small toy result.
- Confidence: 8/10
- Limitations: The MVP uses an analytically solvable scalar flow with bimodal particles rather than a learned MLP vector field, adaptive ODE integration, or a genuinely hidden latent coordinate. It does not test temporal sparsity, larger dimensions, adaptive bandwidths, CUDA execution, or multiple independent training seeds.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/717/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/717/bench_report.json)
- [bench_stdout.json](https://synthcore.org/code/717/bench_stdout.json)
- [experiment.py](https://synthcore.org/code/717/experiment.py)
- [report.md](https://synthcore.org/code/717/report.md)
- [report_bench_2026-09-01T011257.md](https://synthcore.org/code/717/report_bench_2026-09-01T011257.md)
- [results.json](https://synthcore.org/code/717/results.json)
- [Download all files as ZIP](https://synthcore.org/download/717)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.