# Fractional-memory recurrent state

- ID: 2677
- Canonical URL: https://synthcore.org/idea/2677/fractional-memory-recurrent-state
- API JSON: https://synthcore.org/api/idea/2677.json
- API Markdown: https://synthcore.org/api/idea/2677.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.21674](https://arxiv.org/abs/2608.21674)
- Category: architecture
- Solves: accuracy, scalability, sample-efficiency
- ML areas: ssm, rnn, transformer, inference-speedup
- Math tags: fractional-calculus, dynamical-systems, approximation-theory, numerical-analysis
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Construct an efficient recurrent or state-space layer whose impulse response follows Mittag-Leffler relaxation instead of a single exponential. A bank of stable diagonal state channels approximates the long power-law tail, allowing the layer to retain information over widely separated timescales with only \(K\) states per feature.

## Mathematical statement

The paper's temporal building block is \(e_\alpha(t)=E_\alpha(-t^\alpha)=\sum_{n=0}^{\infty}(-t^\alpha)^n/\Gamma(\alpha n+1)\), with fractional order \(\alpha\) and time \(t\). Its stated asymptotic \(e_\alpha(t)\sim t^{-\alpha}/\Gamma(1-\alpha)\) for \(0<\alpha<1\) gives a slowly decaying memory kernel. For a discrete neural sequence \(x_1,\ldots,x_T\), define the desired causal normalized filter \(y_t=\sum_{j=1}^{t}h_{t-j}x_j/\sum_{\ell=0}^{t-1}h_\ell\), where \(h_\ell=E_\alpha(-((\ell+1)/\tau)^\alpha)\), \(\tau>0\) is a characteristic timescale, and \(\ell=t-j\) is the lag. Implement it using a positive exponential mixture \(h_\ell\approx\hat h_\ell=\sum_{k=1}^{K}w_k(1-\rho_k)\rho_k^\ell\). Each component has state \(s_{t,k}=\rho_ks_{t-1,k}+(1-\rho_k)x_t\), and the output is \(y_t=\sum_kw_ks_{t,k}\); stability follows from \(0<\rho_k<1\), while logarithmically spaced \(\rho_k\) approximate the fractional power-law memory.

## Key formulas

- $$e_{\alpha}(t):=E_{\alpha}(-t^{\alpha}):=\sum_{n=0}^{\infty}\frac{(-t^{\alpha})^{n}}{\Gamma(\alpha n+1)}$$
- $$e_{\alpha}(t)\sim \frac{t^{-\alpha}}{\Gamma(1-\alpha)},\qquad t\to\infty,\quad 0<\alpha<1$$
- $$y_t=\frac{\sum_{j=1}^{t}h_{t-j}x_j}{\sum_{\ell=0}^{t-1}h_\ell},\qquad h_\ell=E_{\alpha}\!\left(-\left(\frac{\ell+1}{\tau}\right)^{\alpha}\right)$$
- $$s_{t,k}=\rho_ks_{t-1,k}+(1-\rho_k)x_t,\qquad y_t=\sum_{k=1}^{K}w_ks_{t,k},\qquad \hat h_\ell=\sum_{k=1}^{K}w_k(1-\rho_k)\rho_k^\ell$$

## Implementation notes

Add the module as a drop-in replacement for the temporal mixing block in a small causal Transformer, GRU, or diagonal SSM. For an input sequence tensor \(x\in\mathbb{R}^{T\times B\times d}\), create \(K\) state tensors \(s_k\in\mathbb{R}^{B\times d}\); use \(K=8\), positive weights normalized by softmax, and decay factors \(\rho_k=\exp(-\Delta/\tau_k)\) with logarithmically spaced \(\tau_k\) from 1 to the target context length. At every timestep, compute `s[k] = rho[k] * s[k] + (1-rho[k]) * x[t]`, then `y[t] = sum(w[k] * s[k])`; apply a learned input/output projection and residual connection around the filter. Optionally learn \(\alpha\) and \(\tau\) only through a differentiable initialization map that refits \(w_k,\rho_k\), but begin with fixed stable decays. Fit the bank before training by minimizing squared error between \(\hat h_\ell\) and the paper kernel \(E_\alpha(-((\ell+1)/\tau)^\alpha)\) over logarithmic lags; this fit is the empirical approximation, while the power-law asymptotic is the mathematical motivation. The first cheap experiment should compare this layer with a GRU, a single-exponential SSM, and a standard causal Transformer on synthetic delayed-copy and adding problems at sequence lengths 1k, 4k, and 16k, followed by TinyStories language modeling. Track accuracy as delay/context grows, validation perplexity at equal parameter count, and memory/runtime. Success is materially better delayed-copy accuracy and less perplexity degradation at long context with \(O(KTd)\) compute and constant recurrent state.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a stable PyTorch fractional-memory layer using positive softmax-weighted exponential states with logarithmically spaced time constants, plus numerical verification scripts. The Mittag-Leffler power-law asymptotic matched the predicted 1/sqrt(pi) constant with relative error falling from 0.49% at lag 100 to 0.00005% at lag 1e6, and exponential stability/half-life predictions matched discrete observations within one step. However, bank approximation error improved strongly from K=1 to K=4 but then saturated rather than continuing to improve, and the fractional K=8 filter had higher random-signal MSE than the best single exponential; therefore the full claimed mechanism was not demonstrated as a win.

### Mechanism check

- Verdict: Built a stable PyTorch fractional-memory layer using positive softmax-weighted exponential states with logarithmically spaced time constants, plus numerical verification scripts. The Mittag-Leffler power-law asymptotic matched the predicted 1/sqrt(pi) constant with relative error falling from 0.49% at lag 100 to 0.00005% at lag 1e6, and exponential stability/half-life predictions matched discrete observations within one step. However, bank approximation error improved strongly from K=1 to K=4 but then saturated rather than continuing to improve, and the fractional K=8 filter had higher random-signal MSE than the best single exponential; therefore the full claimed mechanism was not demonstrated as a win.
- Confidence: 9/10
- Limitations: No trained delayed-copy model, GRU, Transformer, language-model, equal-parameter benchmark, GPU benchmark, or alpha values other than 1/2 were tested. The bank fit used fixed logarithmic time constants and a finite horizon, so improved fitting or learned decays remain unexplored.

## Artifacts

- [experiment.py](https://synthcore.org/code/1034/experiment.py)
- [fractional_memory.py](https://synthcore.org/code/1034/fractional_memory.py)
- [report.md](https://synthcore.org/code/1034/report.md)
- [results.json](https://synthcore.org/code/1034/results.json)
- [verify.py](https://synthcore.org/code/1034/verify.py)
- [Download all files as ZIP](https://synthcore.org/download/1034)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.