# Two-Column Non-Markovian Memory Core

- ID: 1427
- Canonical URL: https://synthcore.org/idea/1427/two-column-non-markovian-memory-core
- API JSON: https://synthcore.org/api/idea/1427.json
- API Markdown: https://synthcore.org/api/idea/1427.md
- Verification status: failed_benchmark
- Source: [arXiv:2607.25969](https://arxiv.org/abs/2607.25969)
- Category: architecture
- Solves: memory, accuracy, stability
- ML areas: rnn, ssm, world-model, training-dynamics
- Math tags: dynamical-systems, tensor-decomposition, statistical-mechanics, linear-algebra
- Ratings: usefulness 7/10; difficulty 6/10; novelty 7/10

## Idea description

Replace a Markovian recurrent update with an MPS-valued temporal influence state that couples adjacent pairs of memory sites, mimicking the paper's CDU3 two-column construction. The hidden state retains structured correlations across multiple past time steps while computation remains linear in sequence length and polynomial in the bond dimension, rather than exponential in the memory horizon.

## Mathematical statement

For a local qudit dimension \(q\), the paper's influence matrix is a state \(\lvert L_t\rangle\) on \(2t+1\) temporal sites, each of dimension \(q^2\). Markovian one-column or CDU2 constructions factorize into local or pair tensors, whereas CDU3 connects all temporal sites through two columns while remaining an MPS with bond dimension \(\chi\le q^2\). Transfer this by defining a neural memory tensor with physical feature dimension \(p\), bond dimension \(\chi\), and local matrices \(A^{(x)}\in\mathbb{R}^{\chi\times\chi}\). The sequence functional is \(F_t=\operatorname{tr}(A^{(x_0)}\cdots A^{(x_t)}\rho_0)\). A second pair channel uses \(B^{(x_s,x_{s+1})}\) to couple neighboring temporal sites. Spectral normalization \(\|A^{(x)}\|_2\le\alpha<1\) gives contraction of the linearized recurrence, so perturbations after \(r\) steps satisfy \(\|\delta F_{t+r}\|\le\alpha^r\|\delta F_t\|\).

## Key formulas

- $$\lvert L_t\rangle\in\left(\mathbb{C}^{q^2}\right)^{\otimes(2t+1)},\qquad \lvert L_t\rangle=\sum_{s_0,\ldots,s_{2t}}\operatorname{tr}\!\left(A^{s_0}A^{s_1}\cdots A^{s_{2t}}\right)\lvert s_0s_1\cdots s_{2t}\rangle,\qquad \chi\le q^2.$$
- $$\lvert L_t\rangle_{\mathrm{Markov}}=\bigotimes_{j=1}^{t}\lvert b_j\rangle,\qquad \lvert L_t\rangle_{\mathrm{CDU3}}=\operatorname{MPS}\!\left(A_0,A_1,\ldots,A_{2t}\right).$$
- $$F_t=\operatorname{tr}\!\left(A^{(x_0)}A^{(x_1)}\cdots A^{(x_t)}\rho_0\right),\qquad \|A^{(x)}\|_2\le\alpha<1\ \Longrightarrow\ \|\delta F_{t+r}\|\le\alpha^r\|\delta F_t\|.$$
- $$\mathrm{cost}(t)=O(t\,\chi^2p),\qquad \mathrm{memory}(t)=O(t\,\chi p).$$

## Implementation notes

Integrate the module between the input embedding and prediction head of a small RNN or state-space model. Map each input \(x_t\) to \(e_t\in\mathbb{R}^p\), construct \(A_t=\sum_{a=1}^{p}e_{t,a}A_a\), and update a bond state \(h_{t+1}^{(1)}=A_th_t^{(1)}\). Maintain a second channel representing the two-column construction. Form a pair feature \(z_t=[e_{t-1};e_t]\), construct \(B_t=\sum_b z_{t,b}B_b\), and update \(h_{t+1}^{(2)}=B_th_t^{(2)}\). Normalize both states and feed \([e_t;h_t^{(1)};h_t^{(2)}]\) to an output MLP. Pseudocode is: initialize both bond states; for each time step compute the embedding, build \(A_t\), update the first state, build \(B_t\) from the current and previous embeddings, update the second state, normalize, and predict. The MPS contraction and two-column pair coupling are taken from the paper; spectral normalization and feature projection are adaptations. Estimate each operator norm with one or two power iterations per batch and rescale to target \(\alpha\). First test on permuted-MNIST and synthetic delayed-copy sequences with GRU, diagonal SSM, and factorized Markov-memory baselines. Sweep \(\alpha\in[0.8,1.2]\) and \(\chi\in\{4,8,16\}\). Measure hidden perturbation growth, gradient norms, loss, and wall-clock scaling with sequence length. The quantitative prediction is a stability transition near measured average singular value one: above one, perturbations and gradients grow exponentially with horizon; below one, they decay approximately as \(\alpha^r\). The non-Markovian model should gain most on targets requiring two-step or longer temporal correlations, while runtime remains linear in sequence length.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built an MPS-inspired two-column memory core with single-site transfer matrices, adjacent-pair matrices, spectral normalization utilities, and a delayed XOR learning test. The contraction sweep matched the predicted alpha^r ratios to relative error below 1.6e-15 across alpha=0.6..1.25, including the predicted stability boundary at alpha=1. The pair-channel sweep was exactly linear in its coupling lambda, and the gamma*lambda boundary sweep matched predictions to below 2.3e-15; the toy learner reached higher accuracy than the GRU, though this is only a small full-batch CPU test.

### Mechanism check

- Verdict: Built an MPS-inspired two-column memory core with single-site transfer matrices, adjacent-pair matrices, spectral normalization utilities, and a delayed XOR learning test. The contraction sweep matched the predicted alpha^r ratios to relative error below 1.6e-15 across alpha=0.6..1.25, including the predicted stability boundary at alpha=1. The pair-channel sweep was exactly linear in its coupling lambda, and the gamma*lambda boundary sweep matched predictions to below 2.3e-15; the toy learner reached higher accuracy than the GRU, though this is only a small full-batch CPU test.
- Confidence: 8/10
- Limitations: The CUDA attempt encountered a shared-environment cuDNN/system allocation crash, so the final reproducible run used CPU. No permuted-MNIST, long-horizon benchmark, FLOP-normalized comparison, chi sweep, or multi-seed statistical evaluation was performed; the exact math checks use orthogonal transfer matrices and therefore validate the stated formulas rather than broad learned-model behavior.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/558/bench_report.json)
- [report.md](https://synthcore.org/code/558/report.md)
- [report_bench_2026-08-31T175814.md](https://synthcore.org/code/558/report_bench_2026-08-31T175814.md)
- [results.json](https://synthcore.org/code/558/results.json)
- [stage2_bench.py](https://synthcore.org/code/558/stage2_bench.py)
- [two_column_memory.py](https://synthcore.org/code/558/two_column_memory.py)
- [Download all files as ZIP](https://synthcore.org/download/558)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.