# Random-Travel-Time Temporal Layer

- ID: 2999
- Canonical URL: https://synthcore.org/idea/2999/random-travel-time-temporal-layer
- API JSON: https://synthcore.org/api/idea/2999.json
- API Markdown: https://synthcore.org/api/idea/2999.md
- Verification status: unverified
- Source: [arXiv:2609.01291](https://arxiv.org/abs/2609.01291)
- Category: dynamics
- Solves: stability, accuracy, sample-efficiency
- ML areas: ssm, rnn, training, world-model
- Math tags: pde, probability, dynamical-systems, measure-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace the fixed delay in a temporal layer with a distribution of physically structured delays induced by uncertain transport velocity. The layer aggregates features arriving at several travel times and can use the deterministic mean-velocity path during most training steps, periodically correcting it with stochastic samples.

## Mathematical statement

The transport equation is $\rho_t(x,t)+\lambda\rho_x(x,t)=0$ on $x\in(0,1)$, with inflow condition $\lambda\rho(0,t)=u(t)$. Its characteristic solution for material originating at the boundary is $\rho(x,t)=\lambda^{-1}u(t-x/\lambda)$, so the downstream signal at $x=1$ depends on the delayed time $t-\tau$ with travel time $\tau=1/\lambda$. Let $V$ be a positive random velocity supported on $[\lambda_{\min},\lambda_{\max}]$, let $h_\theta(t)$ be a learned temporal feature signal, and let $\tau(V)=1/V$. The stochastic transport analogue is $z(t)=\mathbb{E}_V[h_\theta(t-\tau(V))]$. A deterministic proxy uses $\bar V=\mathbb{E}[V]$ and $z_{\mathrm{proxy}}(t)=h_\theta(t-1/\bar V)$. The paper's abstract states that velocity-induced irreducible error is bounded linearly in velocity variance and that the additional loss of the deterministic mean-velocity proxy has a higher-order bound; operationally, this motivates using stochastic averaging only when the velocity variance or local temporal curvature is large. For discrete sampled features with timestep $\Delta$, define interpolation $\operatorname{Interp}(h,s)$ at real-valued index $s$ and use $z_k=\sum_{j=1}^m w_j\operatorname{Interp}(h,k-\tau_j/\Delta)$, where $\tau_j=1/v_j$ and $\sum_jw_j=1$.

## Key formulas

- $$\rho_t(x,t)+\lambda\rho_x(x,t)=0,\qquad \lambda\rho(0,t)=u(t)$$
- $$\rho(x,t)=\frac{1}{\lambda}u\!\left(t-\frac{x}{\lambda}\right)\quad\text{when the characteristic originates at the inflow boundary}$$
- $$z(t)=\mathbb{E}_{V}\left[h_\theta\!\left(t-\frac{1}{V}\right)\right],\qquad z_{\mathrm{proxy}}(t)=h_\theta\!\left(t-\frac{1}{\mathbb{E}[V]}\right)$$
- $$z_k\approx\sum_{j=1}^{m}w_j\,\operatorname{Interp}\!\left(h,k-\frac{1}{v_j\Delta}\right),\qquad v_j\sim P_V,\quad \sum_{j=1}^{m}w_j=1$$

## Implementation notes

(1) Integrate this immediately before a temporal mixer in an SSM, GRU, TCN, or lightweight RNN. Given a sequence tensor h of shape [batch, length, channels], create a transport layer that shifts h backward along the time axis by random travel times and averages the shifted tensors; use linear interpolation so that 1/(v Delta) need not be integral. The velocity distribution P_V is a fixed truncated distribution or a learned positive distribution, parameterized by mean and variance, and is shared across channels initially. (2) Pseudocode: sample m velocities v[j] from P_V; for each j compute delay[j]=1/(v[j]*Delta); shifted[j]=linear_interpolate(h, indices-delay[j]); z=sum_j w[j]*shifted[j]; feed z to the original temporal mixer. With probability p_correct, use this stochastic branch; otherwise compute z=linear_interpolate(h, indices-1/(mean_velocity*Delta)), the mean-velocity proxy. Add an optional consistency penalty L_cons=mean(||z_stochastic-stopgrad(z_proxy)||^2) only when velocity variance is below a threshold. (3) Compute from the paper's mathematics the reciprocal delay 1/v and the characteristic shift; estimate no PDE quantities. Empirically estimate whether the proxy is safe by tracking r=||z_stochastic-z_proxy||/(||z_stochastic||+epsilon) on minibatches and increase m or p_correct when r is large. (4) First experiment: train a small GRU or diagonal SSM on permuted-delay sequential MNIST, Copying Memory, and a synthetic advection dataset where targets are generated with velocities sampled from a known interval. Compare fixed-delay, mean-velocity proxy, and m=4 stochastic-delay layers at equal parameter count and report validation loss, robustness under a shifted velocity distribution, and wall-clock cost. Success is lower error under velocity shifts at no more than 20% training overhead, with the proxy retaining most of the stochastic layer's accuracy when the velocity variance is small.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.