# Rank-One Delta Associative Memory

- ID: 1198
- Canonical URL: https://synthcore.org/idea/1198/rank-one-delta-associative-memory
- API JSON: https://synthcore.org/api/idea/1198.json
- API Markdown: https://synthcore.org/api/idea/1198.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2607.20792](https://arxiv.org/abs/2607.20792)
- Category: architecture
- Solves: memory, speedup, sample-efficiency
- ML areas: attention, memory, transformer, inference-speedup
- Math tags: linear-algebra, optimization, dynamical-systems
- Ratings: usefulness 7/10; difficulty 5/10; novelty 5/10

## Idea description

Replace a portion of quadratic key-value attention or an external episodic table with a per-sample matrix fast memory updated by rank-one delta corrections. The memory directly learns a linear key-to-value map and can be carried across sequence segments, providing cheap online adaptation with constant state size per head.

## Mathematical statement

The plastic state consists of per-sample matrices $W_0,W_1\in\mathbb{R}^{B\times H\times D_h\times D_h}$, where $B$ is batch size, $H$ is the number of heads, and $D_h$ is the head dimension. For one head and one stream, the core update is $W^{(t+1)}=W^{(t)}+\beta_t(v_t-W^{(t)}k_t)k_t^{\top}$. Here $W^{(t)}k_t$ is the current prediction for value $v_t$, the residual $v_t-W^{(t)}k_t$ is the correction, and the outer product with $k_t$ changes the map only in the key direction. The update is rank one, so it requires matrix-vector products and an outer product rather than storing all past key-value pairs. The construction supports state persistence across segments and optional detachment at a credit boundary. A multi-head implementation uses one independent $D_h\times D_h$ matrix per batch element and head, with $\beta_t$ either fixed, learned through $\sigma(b_t)$, or generated from the current token.

## Key formulas

- $$W_{0},W_{1}\in\mathbb{R}^{B\times H\times D_{h}\times D_{h}}$$
- $$W^{(t+1)}=W^{(t)}+\beta_{t}\left(v_{t}-W^{(t)}k_{t}\right)k_{t}^{\top}$$
- $$\widehat{v}_{t}=W^{(t)}k_t,\qquad r_t=v_t-\widehat{v}_t,\qquad W^{(t+1)}=W^{(t)}+\beta_t r_t k_t^{\top}$$
- $$\beta_t=\sigma\!\left(b_{\phi}(x_t)\right),\qquad m_t=W^{(t)}k_t$$

## Implementation notes

Add a FastDeltaMemory module before or after attention. Project each input token x_t into per-head key k_t, value v_t, and write-logit b_t using three learned linear projections. Store W with shape [batch, heads, Dh, Dh], initialized to zero or a small scaled orthogonal matrix. For each token, compute prediction m = einsum('bhij,bhj->bhi', W, k), residual r = v - m, and update W in place with W += beta[..., None, None] * r[..., :, None] * k[..., None, :], where beta = sigmoid(b) and beta may be masked for padding tokens. Read the memory as m and combine it with normal attention output through a learned gate or residual addition. For streaming use, preserve W between chunks; for ordinary language modeling, reset it per sequence. To reduce numerical instability, normalize k by max(||k||2, epsilon), clip beta, and use fp32 state even when activations are bf16. The paper supplies the exact delta correction and the per-sample/per-head state ownership; the engineer must measure whether learned projections produce useful keys and values rather than assuming it. First benchmark on 1-8K-token synthetic associative recall with distractors, then on a small WikiText or TinyStories language model. Compare against full causal attention and linear-attention baselines at matched hidden size and FLOPs. Record perplexity or recall at fixed training steps, throughput, peak activation memory, and recurrent-state memory. The expected signal is competitive recall with lower sequence-dependent memory and faster inference on long sequences; failure is random or rapidly saturating W norms, in which case compare fixed beta schedules and stronger key normalization.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built and numerically verified the rank-one delta memory update. The predicted scalar error multiplier, absolute-stability boundary at beta*lambda=2, convergence steps, and orthogonal-key independence matched observations to approximately 1e-15 or exactly on the tested grid, so the mechanism manifested. However, delta memory performed far worse than causal softmax attention on associative recall (MSE 1.1055 vs 8.03e-6), despite using 12x fewer state floats; no task-quality win was observed.

### Mechanism check

- Verdict: Built and numerically verified the rank-one delta memory update. The predicted scalar error multiplier, absolute-stability boundary at beta*lambda=2, convergence steps, and orthogonal-key independence matched observations to approximately 1e-15 or exactly on the tested grid, so the mechanism manifested. However, delta memory performed far worse than causal softmax attention on associative recall (MSE 1.1055 vs 8.03e-6), despite using 12x fewer state floats; no task-quality win was observed.
- Confidence: 9/10
- Limitations: Only a synthetic fixed-projection associative-recall task was tested; no learned projections, language model, WikiText/TinyStories, multi-head implementation, streaming segments, throughput, GPU memory, or matched-FLOP training comparison was evaluated. The attention baseline had direct access to all stored write keys and values, while delta memory compressed them into a fixed matrix.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.31255
- Paired wins: 2/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_delta_memory.py](https://synthcore.org/code/485/bench_delta_memory.py)
- [bench_report.json](https://synthcore.org/code/485/bench_report.json)
- [experiment.py](https://synthcore.org/code/485/experiment.py)
- [local_sequence_bench.py](https://synthcore.org/code/485/local_sequence_bench.py)
- [report.md](https://synthcore.org/code/485/report.md)
- [report_bench_2026-08-31T154951.md](https://synthcore.org/code/485/report_bench_2026-08-31T154951.md)
- [results.json](https://synthcore.org/code/485/results.json)
- [Download all files as ZIP](https://synthcore.org/download/485)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.