# Explicit Multi-Timescale Memory Bank

- ID: 1766
- Canonical URL: https://synthcore.org/idea/1766/explicit-multi-timescale-memory-bank
- API JSON: https://synthcore.org/api/idea/1766.json
- API Markdown: https://synthcore.org/api/idea/1766.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.04028](https://arxiv.org/abs/2608.04028)
- Category: architecture
- Solves: accuracy, stability, sample-efficiency
- ML areas: rnn, ssm, memory, world-model
- Math tags: dynamical-systems, spectral-theory, linear-algebra, numerical-analysis
- Ratings: usefulness 7/10; difficulty 4/10; novelty 7/10

## Idea description

Use a bank of damped rotational state channels with a deliberately spread decay spectrum, allowing one recurrent layer to represent short, medium, and long temporal dependencies without relying on a single learned spectral radius. Concatenate the channels and train a readout or downstream nonlinear head to select the appropriate memory timescale.

## Mathematical statement

For mode j, the scalar contraction factor is r_j = exp(-gamma_j), so an unforced linear mode retains information for a characteristic timescale approximately tau_j = 1/gamma_j when gamma_j is small. Its state rotates by omega_j radians per step while decaying by r_j. The paper's ensemble construction uses parallel members z_t^(m), each obeying z_{t+1}^(m) = tanh(W_m z_t^(m) + W_in^(m) u_t + b_m), followed by concatenation z_t = col_m z_t^(m). The transferable mechanism is to prescribe or learn a log-spaced set of gamma_j values, for example gamma_j in [gamma_min, gamma_max], so the memory bank covers exponentially separated horizons. The contraction bound for every member is ||W_m||_2 = exp(-min_j gamma_mj), while all modes remain bounded because exp(-gamma_j) <= 1. Frequencies omega_j control phase diversity independently of the memory horizon and can be sampled over a chosen interval such as [0, pi].

## Key formulas

- $$B_j=e^{-\gamma_j}R(\omega_j),\qquad r_j=|\lambda_j|=e^{-\gamma_j},\qquad \tau_j\approx\frac{1}{\gamma_j}$$
- $$\gamma_j=\exp\!\left(\log\gamma_{\min}+\frac{j-1}{K-1}\left(\log\gamma_{\max}-\log\gamma_{\min}\right)\right),\qquad j=1,\ldots,K$$
- $$z_{t+1}^{(m)}=\tanh\!\left(W_m z_t^{(m)}+W_{\mathrm{in}}^{(m)}u_t+b_m\right),\qquad z_t=\operatorname{col}_{m=1}^{M}z_t^{(m)}$$
- $$\|W_m\|_2=e^{-\min_j\gamma_{mj}}<1$$

## Implementation notes

Integrate this as a parallel recurrent feature extractor before a linear head, MLP head, or attention pooling layer. Split a fixed state budget d across M members or use one member containing K mode pairs. Initialize the decay rates log-uniformly over the target sequence horizons: choose tau_min and tau_max in timesteps, set gamma_min = 1/tau_max and gamma_max = 1/tau_min, then use the displayed geometric interpolation to obtain gamma_j. Sample omega_j uniformly in [0, pi] or use a deterministic frequency grid; construct every block B_j exactly from cosine and sine, and optionally apply a fixed orthogonal Q to mix coordinates. At each timestep, update all members in parallel, concatenate their states, and feed the result to the prediction head. Keep the bank fixed for the first experiment so improvements cannot be attributed to extra recurrent optimization; then unfreeze only gamma_j using gamma_j = softplus(rho_j) to test adaptive timescales. Compute the decay spectrum analytically and report the effective horizon of every mode; estimate empirical memory by linear-probe accuracy for reconstructing u_{t-k} at delays k. Start with a 128-state model on copy-memory, delayed XOR, NARMA-20, and Mackey-Glass, comparing against a same-width vanilla RNN and an orthogonal RNN. The expected signal is a flatter delay-reconstruction curve across short and long k, lower NARMA-20 error, and less degradation as required delay exceeds the median baseline memory horizon.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built an explicit parallel damped-rotation memory bank with log-spaced decay rates and a delayed-input linear probe. The three quantitative mechanism predictions were confirmed: decay-log slope matched -gamma to below 6e-17 absolute error, integer e-fold horizons matched 1/gamma exactly for four rates, and log-horizon spacing matched to 2.2e-16. In the toy probe, the bank improved long-delay MSE substantially at delays 32 and 64 (0.532 vs 0.897 and 0.867 vs 0.943), though the single-timescale baseline was better at several short delays, so the practical gain is horizon-dependent rather than universal.

### Mechanism check

- Verdict: Built an explicit parallel damped-rotation memory bank with log-spaced decay rates and a delayed-input linear probe. The three quantitative mechanism predictions were confirmed: decay-log slope matched -gamma to below 6e-17 absolute error, integer e-fold horizons matched 1/gamma exactly for four rates, and log-horizon spacing matched to 2.2e-16. In the toy probe, the bank improved long-delay MSE substantially at delays 32 and 64 (0.532 vs 0.897 and 0.867 vs 0.943), though the single-timescale baseline was better at several short delays, so the practical gain is horizon-dependent rather than universal.
- Confidence: 9/10
- Limitations: Only a linear reservoir and linear readout on Gaussian delayed reconstruction were tested; no tanh recurrence, learned gamma values, nonlinear tasks such as NARMA/Mackey-Glass, orthogonal or trained vanilla RNN baseline, multiple seeds, FLOP/throughput measurement, or GPU implementation was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/662/bench_report.json)
- [memory_bank_experiment.py](https://synthcore.org/code/662/memory_bank_experiment.py)
- [report.md](https://synthcore.org/code/662/report.md)
- [report_bench_2026-08-31T211728.md](https://synthcore.org/code/662/report_bench_2026-08-31T211728.md)
- [results.json](https://synthcore.org/code/662/results.json)
- [stage2_memory_bank.py](https://synthcore.org/code/662/stage2_memory_bank.py)
- [Download all files as ZIP](https://synthcore.org/download/662)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.