Signature-memory neural CDE

Integrate this at the encoder or latent-dynamics component of a sequence model, before the prediction head. Inputs are an irregular or regularly sampled path $x_0,\ldots,x_T\in\mathbb{R}^{d_x}$, an initial hidden state $h_0$, signature depth $N$, and time step $\Delta t$. Augment each input with time, observation mask, and optional confidence channels. Maintain tensors $S^{(0)}=1$ and $S^{(k)}=0$ for $k>0$. On each interval compute $\Delta x_t=x_{t+1}-x_t$. Using the old signature levels, update the truncated signature by $S^{(k)}_{t+1}=S^{(k)}_t+S^{(k-1)}_t\otimes\Delta x_t$ for $k=1,\ldots,N$; perform the updates simultaneously or loop from high degree to low degree. Concatenate $S^{\leq N}$ into $s_t$, optionally project it to dimension $d_s$ with a learned linear layer, and evaluate MLPs $g_\theta(h_t,s_t)$ and $B_\theta(h_t,s_t)$. Use $h_{t+1}=h_t+\Delta t\,g_\theta(h_t,s_t)+B_\theta(h_t,s_t)\Delta x_t$, with Heun or midpoint integration as an improved solver. Normalize each signature degree, clip gradients, and initialize a residual gate near zero. The paper provides the signature state-space construction; signature values and solver errors are estimated numerically, while $g_\theta$ and $B_\theta$ are learned by backpropagation. Start with adding, permuted sequential MNIST, and a synthetic long-memory regression $y_T=\int_0^T x_t x_{t-\tau}\,dt$. Compare GRU, standard neural CDE, and signature-memory CDE at equal hidden width and FLOPs. Sweep $N\in\{1,2,3,4\}$ and report MSE or accuracy, long-horizon extrapolation error, gradient-norm variance, and robustness to irregular sampling. Success is lower error at equal parameter count, monotonic improvement with useful signature depth, and fewer exploding or vanishing-gradient failures.