# Fading-Memory Contractive Recurrent Block

- ID: 3058
- Canonical URL: https://synthcore.org/idea/3058/fading-memory-contractive-recurrent-block
- API JSON: https://synthcore.org/api/idea/3058.json
- API Markdown: https://synthcore.org/api/idea/3058.md
- Verification status: unverified
- Source: [arXiv:2609.02454](https://arxiv.org/abs/2609.02454)
- Category: architecture
- Solves: stability, accuracy
- ML areas: rnn, ssm, training-dynamics
- Math tags: dynamical-systems, control-theory, functional-analysis, spectral-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace an unrestricted recurrent state with a current input-driven state coupled to a fading average of its own history. The memory coupling provides a distributed dissipative channel, while the recurrent Jacobian is constrained below a measurable contraction boundary. This creates a recurrent or state-space block whose long-horizon sensitivity can be tested against explicit characteristic roots.

## Mathematical statement

Let h(t) be a hidden state and F(h,u) an instantaneous recurrent vector field. Use \(\dot h=F(h,u)+\kappa\int_0^\infty\mu(s)[h(t-s)-h(t)]ds\). For two trajectories with identical input, their difference e satisfies \(\dot e=A(t)e+\kappa\int\mu(s)[e(t-s)-e(t)]ds\), where A(t) is a mean-value Jacobian of F. For an exponential kernel \(\mu(s)=\beta e^{-\beta s}\), introduce \(m_t=\beta(h-m)\) and obtain \(\dot h=F(h,u)+\kappa(m-h)\). For a scalar linear mode \(F(h)=ah\), the continuous-time characteristic polynomial is \(r^2+(\beta+\kappa-a)r-\beta a=0\). The actual nonlinear design must constrain the closed-loop Jacobian, for example using spectral normalization, negative leakage, or a contractive activation.

## Key formulas

- $$\dot h(t)=F(h(t),u(t))+\kappa\int_0^\infty\mu(s)\bigl[h(t-s)-h(t)\bigr]ds,\qquad \mu(s)=\beta e^{-\beta s}.$$
- $$m_t=\beta(h-m),\qquad \dot h=F(h,u)+\kappa(m-h).$$
- $$V(e)=\frac12\lVert e(t)\rVert^2+\frac{\kappa}{2}\int_0^\infty\mu(s)\lVert e(t)-e(t-s)\rVert^2ds.$$
- $$F(h)=ah\quad\Longrightarrow\quad r^2+(\beta+\kappa-a)r-\beta a=0.$$

## Implementation notes

1. Integration point: implement a recurrent cell or continuous-time state-space layer with hidden state h and one additional state m of equal dimension. Use a gated or spectrally normalized recurrent map F, then add kappa times (m minus h) before discretization. With step size Delta t, use h_new equal to h plus Delta t times [F(h,u) plus kappa(m-h)], and m_new equal to exp(-beta Delta t) times m plus [one minus exp(-beta Delta t)] times h_new. 2. Pseudocode: compute q equal to F_theta(h,u); compute h_new from the Euler equation; compute m_new from the exact exponential filter; return both states. Initialize m equal to h. 3. The paper-derived components are the history augmentation and the energy functional. Estimate the local Jacobian gain with one to three power iterations on the Jacobian of F with respect to h per batch; do not assume the weight spectral norm exactly equals the nonlinear Jacobian norm. Log perturbation growth, hidden-state norms, and the estimated largest real characteristic root. 4. First cheap experiment: compare a vanilla tanh RNN, a spectrally normalized RNN, and this fading-memory cell on the synthetic copy task and permuted-MNIST. Run two identical inputs from states separated by a perturbation of size 10^-5. On a scalar linearized cell, numerically calculate the roots of the stated polynomial and compare them with the observed slope of log perturbation norm versus time. The predicted signature is a transition from exponential amplification to decay as the recurrent gain crosses the root-zero boundary; the observed transition should be within 20 percent of the predicted gain threshold.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.