# Signature-memory neural CDE

- ID: 883
- Canonical URL: https://synthcore.org/idea/883/signature-memory-neural-cde
- API JSON: https://synthcore.org/api/idea/883.json
- API Markdown: https://synthcore.org/api/idea/883.md
- Verification status: unverified
- Source: [arXiv:2607.13886](https://arxiv.org/abs/2607.13886)
- Category: architecture
- Solves: accuracy, stability, sample-efficiency
- ML areas: rnn, world-model, training-dynamics
- Math tags: stochastic-processes, dynamical-systems, tensor-decomposition
- Ratings: usefulness 6/10; difficulty 5/10; novelty 5/10

## Idea description

Replace an unconstrained recurrent memory with a truncated path-signature state that is updated continuously from the input control path. Feed this structured state to a learned vector field, allowing the model to represent path-dependent dynamics through iterated integrals of the entire history rather than only the latest hidden state.

## Mathematical statement

The paper formulates a path-dependent controlled differential equation as $\mathrm{d}y_t=f(y|_{[0,t]})\,\mathrm{d}x_t$, where $x:[T_0,T_1]\to\mathbb{R}^{d_x}$ is the control path, $y|_{[0,T_0]}=w$ is the initial history, $y|_{[0,t]}$ is the complete state history up to time $t$, and $f$ is a functional of that history. The transfer replaces the history argument by a truncated signature $S_t^{\leq N}(x)$, consisting of iterated integrals of $x$ through degree $N$. Define $S_t^{(0)}=1$, $S_t^{(1)}=x_t-x_{T_0}$, and recursively $\mathrm{d}S_t^{(k)}=S_t^{(k-1)}\otimes\mathrm{d}x_t$ for $k=1,\ldots,N$, where $\otimes$ is the tensor product and $S_t^{(k)}\in(\mathbb{R}^{d_x})^{\otimes k}$. The neural approximation uses a learned vector field $g_\theta$ on the signature state: $\mathrm{d}h_t=g_\theta(h_t,S_t^{\leq N})\,\mathrm{d}t+B_\theta(h_t,S_t^{\leq N})\,\mathrm{d}x_t$. The signature lift is the finite-dimensional analogue of a Sig-CDE/infinite-dimensional state-space lift; increasing $N$ increases the order of history interactions, while the learned vector field handles task-specific dynamics.

## Key formulas

- $$\mathop{\mathrm{d}}y_t=f(y|_{[0,t]})\,\mathop{\mathrm{d}}x_t,\qquad t\in[T_0,T_1],\qquad y|_{[0,T_0]}=w.$$
- $$S_t^{(0)}=1,\qquad \mathrm{d}S_t^{(k)}=S_t^{(k-1)}\otimes\mathrm{d}x_t,\quad k=1,\ldots,N,$$
- $$\mathrm{d}h_t=g_\theta(h_t,S_t^{\leq N})\,\mathrm{d}t+B_\theta(h_t,S_t^{\leq N})\,\mathrm{d}x_t,\qquad S_t^{\leq N}=\bigoplus_{k=0}^{N}S_t^{(k)}.$$

## Implementation notes

Integrate this at the encoder or latent-dynamics component of a sequence model, before the prediction head. Inputs are an irregular or regularly sampled path $x_0,\ldots,x_T\in\mathbb{R}^{d_x}$, an initial hidden state $h_0$, signature depth $N$, and time step $\Delta t$. Augment each input with time, observation mask, and optional confidence channels. Maintain tensors $S^{(0)}=1$ and $S^{(k)}=0$ for $k>0$. On each interval compute $\Delta x_t=x_{t+1}-x_t$. Using the old signature levels, update the truncated signature by $S^{(k)}_{t+1}=S^{(k)}_t+S^{(k-1)}_t\otimes\Delta x_t$ for $k=1,\ldots,N$; perform the updates simultaneously or loop from high degree to low degree. Concatenate $S^{\leq N}$ into $s_t$, optionally project it to dimension $d_s$ with a learned linear layer, and evaluate MLPs $g_\theta(h_t,s_t)$ and $B_\theta(h_t,s_t)$. Use $h_{t+1}=h_t+\Delta t\,g_\theta(h_t,s_t)+B_\theta(h_t,s_t)\Delta x_t$, with Heun or midpoint integration as an improved solver. Normalize each signature degree, clip gradients, and initialize a residual gate near zero. The paper provides the signature state-space construction; signature values and solver errors are estimated numerically, while $g_\theta$ and $B_\theta$ are learned by backpropagation. Start with adding, permuted sequential MNIST, and a synthetic long-memory regression $y_T=\int_0^T x_t x_{t-\tau}\,dt$. Compare GRU, standard neural CDE, and signature-memory CDE at equal hidden width and FLOPs. Sweep $N\in\{1,2,3,4\}$ and report MSE or accuracy, long-horizon extrapolation error, gradient-norm variance, and robustness to irregular sampling. Success is lower error at equal parameter count, monotonic improvement with useful signature depth, and fewer exploding or vanishing-gradient failures.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.