Delay-Robust Cooperative Recurrent Cell

Integrate the cell into an RNN, neural ODE discretization, or selective state-space model used for long-sequence prediction. Maintain two states: the current hidden state \(h_t\) and a filtered communicated state \(\bar h_{t-1}\). At each step sample a delay \(\tau_t\in[0,\tau_{\max}]\), retrieve the hidden state from a buffer at time \(t-\tau_t\), and update the filter using \(\bar h_t=(1-2\Delta/t_d)\bar h_{t-1}+(4\Delta/t_d)u_t\), where \(u_t\) is the delayed predecessor projection and \(\Delta\) is the integration step. Compute \(h_t=\phi(W_hh_{t-1}+W_c\bar h_{t-1}+W_xx_t+b)\). During training, estimate the largest singular value of the augmented Jacobian \(J_t\) by one or two power iterations and add \(\lambda\max(0,\sigma_{\max}(J_t)-1)^2\), or enforce spectral normalization on \(W_h,W_c\). The paper provides the delay-filter construction and stability rationale; the neural Jacobian bound is an empirical adaptation. First experiment: train a GRU baseline and this delayed cooperative cell on sequential MNIST or the copy-memory task, with train-time delays of 0, 5, and 20 steps and evaluation at 100-step horizons. Measure hidden perturbation growth and rollout error. The predicted signature is a delay-dependent stability boundary: once the estimated largest Jacobian singular value exceeds 1, perturbations should grow exponentially with horizon; below 1, perturbations should decay approximately as \(\sigma^T\), where \(\sigma\) is the measured average singular value.