# Slow-Mode Adaptive Memory Gate

- ID: 1356
- Canonical URL: https://synthcore.org/idea/1356/slow-mode-adaptive-memory-gate
- API JSON: https://synthcore.org/api/idea/1356.json
- API Markdown: https://synthcore.org/api/idea/1356.md
- Verification status: unverified
- Source: [arXiv:2607.24310](https://arxiv.org/abs/2607.24310)
- Category: dynamics
- Solves: accuracy, stability, speedup
- ML areas: ssm, rnn, inference-speedup, training-dynamics
- Math tags: dynamical-systems, spectral-theory, linear-algebra, numerical-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Use the spectral time constant of a memory operator to decide when a sequence layer should retain state, refresh it, or bypass expensive long-memory computation. A mode with eigenvalue near one is treated as valuable long memory, while unstable modes are suppressed, yielding an adaptive-computation mechanism driven by operator dynamics rather than token magnitude alone.

## Mathematical statement

For each memory expert, the lifted state is x_t=[q_t;z_t] and evolves linearly as x_{t+1}=M x_t plus input and bias, where M=[[A,B],[C,D]]. The relevant diagnostic is the magnitude of the slowest eigenvalue lambda_star=max_i|lambda_i(M)|. If 0<lambda_star<1, repeated application decays as lambda_star^k and has characteristic decay time tau=-1/log(lambda_star). The exact memory contribution at lag k is K_k q_{t-k}=B D^{k-1} C q_{t-k}; therefore large tau corresponds to slowly decaying historical influence, while lambda_star>=1 indicates a potentially unstable lifted recurrence. The adaptation uses tau as a detached control signal, not as a claimed universal tipping-point clock: retain or refresh memory according to a learned threshold on tau and current uncertainty.

## Key formulas

- $$q_{n+1}=Aq_n+Bz_n+Fu_n+b_0+\epsilon_{n+1},\qquad z_{n+1}=Cq_n+Dz_n,$$
- $$K_k=BD^{k-1}C,\qquad k\geq 1,$$
- $$\lambda_\star=\max_i|\lambda_i(M)|,\qquad M=\begin{pmatrix}A&B\\C&D\end{pmatrix},\qquad \tau=-\frac{1}{\log(\max(\lambda_\star,\varepsilon))},$$
- $$r_t=\sigma\!\left(w_0+w_1\log(1+\tau_t)+w_2\,\|q_t-q_{t-1}\|_2\right),\qquad z_{t+1}\leftarrow r_t(Cq_t+Dz_t)+(1-r_t)z_t.$$

## Implementation notes

Add this mechanism to a recurrent or SSM layer that already maintains a state z_t. At each chunk boundary, rather than recomputing an eigendecomposition, estimate a conservative spectral scale s_t for M using 5 to 10 power iterations; for a fixed learned M this can be cached and updated only once every 128 tokens or once per optimizer step. Set tau_t=-1/log(max(s_t,1e-4)) and detach it from backpropagation. Compute a retention scalar r_t=sigmoid(w0+w1 log(1+tau_t)+w2 delta_t), where delta_t=||q_t-q_{t-1}||_2 normalized by a running RMS. Update the memory as z_new=r_t(C q_t+D z_t)+(1-r_t)z_t. For streaming inference, skip the full memory update when r_t<0.05 and refresh or reset the state when delta_t exceeds a learned threshold; for training, still compute the dense update so gradients remain comparable, or use straight-through skipping. Enforce D stability with D=rho tanh(Draw)/sigma_max(Draw), rho<0.99, and add a penalty if the estimated full-block scale s_t exceeds 0.99. The exact paper-derived computation is the state transition and its exponential decay interpretation; the threshold, retention weights, and spectral estimate are learned or empirical. First evaluate on WikiText-103 or PG19 using a 2-layer 512-width recurrent or SSM language model, against the same model with always-on state updates. Report perplexity at context lengths 512, 2048, and 8192, tokens per second, fraction of skipped updates, and hidden-state norm. A successful result should preserve or improve long-context perplexity while reducing recurrent update FLOPs and eliminating instability caused by near-unit or super-unit modes.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.