Unverified 2026

Laplace-Margin Regularized Depression RNN

Implementation & benchmark of arXiv:2607.16017 — On large networks of integrate-and-fire neurons with short-term synaptic plasticity

Usefulness6/10

Difficulty6/10

Novelty7/10

Dynamics Stability Accuracy Generalization Rnn Ssm Training dynamics Regularization Dynamical systems Control theory Stochastic processes

Source paper: On large networks of integrate-and-fire neurons with short-term synaptic plasticity arXiv:2607.16017 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Augment a recurrent or state-space layer with a bounded synaptic-depression variable that multiplicatively reduces recurrent transmission after activity. During training, estimate the layer's impulse-response transform and penalize characteristic roots approaching the unstable half-plane. This directly targets slow oscillations and exploding recurrent feedback rather than relying only on gradient clipping.

Formulas

$$\frac{d}{dt}\begin{bmatrix}v(t)\\x(t)\end{bmatrix}=\begin{bmatrix}b(v(t))+\alpha\\-f(v(t))x(t)\end{bmatrix},\qquad v\leftarrow0,\quad x\leftarrow(1-U)x\ \text{at a spike}.$$

$$\dot y_3(t)=1+[b'(y_1(t))-z]y_3(t),\qquad \dot y_4(t)=f(y_1(t))+[b'(y_1(t))-z]y_4(t),\qquad y_1(t)=\varphi_t^\alpha(0).$$

$$\widehat\Theta(z)\approx\Delta t\sum_{k=0}^{K}e^{-z k\Delta t}\Theta(k\Delta t),\qquad D(z)=1-g\widehat\Theta(z).$$

$$\mathcal L_{\mathrm{stab}}=\sum_{z\in\mathcal G}\operatorname{softplus}\!\left(\log\frac{|g\widehat\Theta(z)|}{1-\epsilon}\right),\qquad |g\widehat\Theta(a+i\omega)|\le1-\epsilon\ \text{for }a\ge-\rho.$$

Mathematical statement

The model uses a membrane state with deterministic drift and a depression state that decays at activity-dependent rate: $\dot v=b(v)+\alpha$ and $\dot x=-f(v)x$, where $v$ is the recurrent state, $x\in[0,1]$ is available synaptic efficacy, $b$ is intrinsic drift, $f(v)\ge0$ is an activation or firing rate, and $\alpha$ is external plus mean recurrent drive. At an event, the paper applies the regenerative updates $v\leftarrow0$ and $x\leftarrow(1-U)x$, with depression strength $U\in[0,1]$. The linear response is represented by auxiliary ODEs including $\dot y_3=1+[b'(y_1)-z]y_3$ and $\dot y_4=f(y_1)+[b'(y_1)-z]y_4$, where $z\in\mathbb C$ is the Laplace variable and $y_1=\varphi_t^\alpha(0)$ is the unperturbed trajectory. The paper's stability criterion uses zeros of the Laplace transform $\widehat\Theta_\alpha(z)$ of a linear-response kernel. For a neural layer with scalar recurrent gain $g$, define $D(z)=1-g\widehat\Theta(z)$ and require $D(z)\neq0$ for $\operatorname{Re}z\ge-\rho$, where $\rho>0$ is a desired margin. For vector recurrence, monitor $\sigma_{\max}(W\widehat\Theta(z))<1$. Estimate the transform from a finite impulse response using $\widehat\Theta(z)\approx\Delta t\sum_k e^{-zk\Delta t}\Theta(k\Delta t)$.

Implementation notes

Integrate this at the recurrent-state update of a GRU-like RNN or diagonal state-space layer, rather than at the optimizer. For every channel maintain a hidden state $h_t$ and depression variable $q_t\in[0,1]$. Use the discrete dynamics $r_t=\operatorname{softplus}(W_{in}u_t+W_{rec}(q_t\odot h_t)+b)$, $q_{t+1}=\operatorname{clip}(q_t+\Delta t[-\lambda_q q_t-r_q r_t q_t],0,1)$, and $h_{t+1}=h_t+\Delta t[b(h_t)+W_{rec}(q_t\odot r_t)+W_{in}u_t]$. Optionally approximate regenerative events by resetting a unit when $r_t$ exceeds a threshold: $h_t\leftarrow0$, $q_t\leftarrow(1-U)q_t$. Every $M$ optimizer steps, freeze the parameters and estimate the impulse response $\Theta(k\Delta t)$ by injecting a small perturbation into the recurrent drive, recording the activity perturbation, and subtracting a no-impulse rollout. Evaluate $\widehat\Theta(a+i\omega)=\Delta t\sum_k e^{-(a+i\omega)k\Delta t}\Theta(k\Delta t)$ on a grid with $a\in[-\rho,0]$ and frequencies through Nyquist. For scalar recurrence use $D(z)=1-g\widehat\Theta(z)$; for matrix recurrence estimate $\sigma_{\max}(W_{rec}\widehat\Theta(z))$. Add the softplus penalty or reject parameter updates violating the margin. The response transform and grid are estimated empirically; the depression recurrence and Laplace stability construction are the transplanted mathematics. First test a one-layer tanh or softplus RNN on sequential MNIST and the adding problem, against the same RNN with spectral normalization and gradient clipping. Measure validation accuracy, gradient-norm variance, hidden-state autocorrelation, loss spikes, and the largest observed response singular value. Success means fewer long-period oscillations and stable training at recurrent gains where the baseline diverges, without increasing parameter count substantially.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.