△ Mechanism confirmed, baseline not beaten 2026

Rank-One Delta Associative Memory

Implementation & benchmark of arXiv:2607.20792 — Memoir: Should a Model Write to Its Memory While It Thinks?

Usefulness7/10

Difficulty5/10

Novelty5/10

Architecture Memory Speedup Sample efficiency Attention Memory Transformer Inference speedup Linear algebra Optimization Dynamical systems

Source paper: Memoir: Should a Model Write to Its Memory While It Thinks? arXiv:2607.20792 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a portion of quadratic key-value attention or an external episodic table with a per-sample matrix fast memory updated by rank-one delta corrections. The memory directly learns a linear key-to-value map and can be carried across sequence segments, providing cheap online adaptation with constant state size per head.

Formulas

$$W_{0},W_{1}\in\mathbb{R}^{B\times H\times D_{h}\times D_{h}}$$

$$W^{(t+1)}=W^{(t)}+\beta_{t}\left(v_{t}-W^{(t)}k_{t}\right)k_{t}^{\top}$$

$$\widehat{v}_{t}=W^{(t)}k_t,\qquad r_t=v_t-\widehat{v}_t,\qquad W^{(t+1)}=W^{(t)}+\beta_t r_t k_t^{\top}$$

$$\beta_t=\sigma\!\left(b_{\phi}(x_t)\right),\qquad m_t=W^{(t)}k_t$$

Mathematical statement

The plastic state consists of per-sample matrices $W_0,W_1\in\mathbb{R}^{B\times H\times D_h\times D_h}$, where $B$ is batch size, $H$ is the number of heads, and $D_h$ is the head dimension. For one head and one stream, the core update is $W^{(t+1)}=W^{(t)}+\beta_t(v_t-W^{(t)}k_t)k_t^{\top}$. Here $W^{(t)}k_t$ is the current prediction for value $v_t$, the residual $v_t-W^{(t)}k_t$ is the correction, and the outer product with $k_t$ changes the map only in the key direction. The update is rank one, so it requires matrix-vector products and an outer product rather than storing all past key-value pairs. The construction supports state persistence across segments and optional detachment at a credit boundary. A multi-head implementation uses one independent $D_h\times D_h$ matrix per batch element and head, with $\beta_t$ either fixed, learned through $\sigma(b_t)$, or generated from the current token.

Implementation notes

Add a FastDeltaMemory module before or after attention. Project each input token x_t into per-head key k_t, value v_t, and write-logit b_t using three learned linear projections. Store W with shape [batch, heads, Dh, Dh], initialized to zero or a small scaled orthogonal matrix. For each token, compute prediction m = einsum('bhij,bhj->bhi', W, k), residual r = v - m, and update W in place with W += beta[..., None, None] * r[..., :, None] * k[..., None, :], where beta = sigmoid(b) and beta may be masked for padding tokens. Read the memory as m and combine it with normal attention output through a learned gate or residual addition. For streaming use, preserve W between chunks; for ordinary language modeling, reset it per sequence. To reduce numerical instability, normalize k by max(||k||2, epsilon), clip beta, and use fp32 state even when activations are bf16. The paper supplies the exact delta correction and the per-sample/per-head state ownership; the engineer must measure whether learned projections produce useful keys and values rather than assuming it. First benchmark on 1-8K-token synthetic associative recall with distractors, then on a small WikiText or TinyStories language model. Compare against full causal attention and linear-attention baselines at matched hidden size and FLOPs. Record perplexity or recall at fixed training steps, throughput, peak activation memory, and recurrent-state memory. The expected signal is competitive recall with lower sequence-dependent memory and faster inference on long sequences; failure is random or rapidly saturating W norms, in which case compare fixed beta schedules and stronger key normalization.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Inconclusive

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and numerically verified the rank-one delta memory update. The predicted scalar error multiplier, absolute-stability boundary at beta*lambda=2, convergence steps, and orthogonal-key independence matched observations to approximately 1e-15 or exactly on the tested grid, so the mechanism manifested. However, delta memory performed far worse than causal softmax attention on associative recall (MSE 1.1055 vs 8.03e-6), despite using 12x fewer state floats; no task-quality win was observed.

Agent confidence: 9/10

Baseline: Causal softmax attention: MSE 8.03e-6, cosine 0.999999, 12,288 stored write floats
Idea: Rank-one delta memory: MSE 1.1055, cosine 0.2875, 1,024 state floats; observed stability boundary beta*lambda=2.000; multiplier maximum error 2.22e-15; predicted and observed convergence steps matched at 88, 42, 19, and 19

Limitations: Only a synthetic fixed-projection associative-recall task was tested; no learned projections, language model, WikiText/TinyStories, multi-head implementation, streaming segments, throughput, GPU memory, or matched-FLOP training comparison was evaluated. The attention baseline had direct access to all stored write keys and values, while delta memory compressed them into a fixed matrix.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.20792) Sep 2, 2026

No significant difference vs. the tuned baseline (+3.7%, p=0.3125, wins 2 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 1.0789
Idea mean: 1.1184
Effect (Δ): 0.0395 (+3.7%; negative = idea better)
Wins: 2 / 8 paired seeds
p-value: 0.3125 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Inconclusive
Baseline tuning: swept over 3 configs

Limitations:

The official run used a reduced but valid equal-budget configuration of 400 training samples, 150 test samples, and 3 epochs to fit the 40-minute limit. The idea sweep used beta=0.5 and the shared learning-rate union [0.0015, 0.003, 0.006], while the baseline was tuned with sweep_baseline on seeds 0-3 and fully evaluated on seeds 0-7. Throughput, peak activation memory, and long-stream persistence were not measured.

How to run: python3 bench_delta_memory.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

On the registered sequence track, the matched transformer_tiny baseline achieved lower test MSE than the transformer with rank-one delta memory. The mechanism was active in trained models, with finite state norms and nonzero update/read activity, but this did not translate into a task-metric win; therefore worked=false.

Baseline: Tuned transformer_tiny, lr=0.0015: mean test MSE 1.0789237320 across 8 seeds.
Delta: Idea minus baseline = +0.0394928679 MSE; positive means worse because lower is better.
Idea: Best tested delta-memory transformer configuration: lr=0.0015, beta=0.5: mean test MSE 1.1184165999 across 8 seeds.
P value: See bench_report.comparison.p_value in the saved official make_report JSON.

Benchmark result