Memoir: Should a Model Write to Its Memory While It Thinks?
arXiv:2607.20792
2026
Architecture
2 ideas extracted · analyzed Aug 30, 2026
What the math gives to ML
The paper provides a concrete fast-weight construction in which each sample carries matrix-valued plastic state and updates it with a rank-one delta rule. Its most transferable result is an experimentally supported dynamical-systems insight: allowing an iterative latent reasoner to both read and rewrite the same fast memory at every pondering step can slow learning substantially, even when the final converged capability is unchanged. A practical adaptation is to separate read-only internal reasoning from explicit memory-write phases, or to gate writes by confidence and halt state. The rank-one update also offers a compact associative-memory module that can be benchmarked against attention on streaming recall tasks.
Ideas from this paper
△ Mechanism confirmed, baseline not beaten
2026
Use fast memory as read-only scratch state during the internal pondering iterations of a recurrent block, and apply memory writes only after the latent computation has halted or crossed a write gate. This prevents the transition operator from changing while it is being iterated, reducing self-corruption of the evidence used for subsequent reasoning.
Useful8/10
Difficulty4/10
Novelty6/10
△ Mechanism confirmed, baseline not beaten
2026
Replace a portion of quadratic key-value attention or an external episodic table with a per-sample matrix fast memory updated by rank-one delta corrections. The memory directly learns a linear key-to-value map and can be carried across sequence segments, providing cheap online adaptation with constant state size per head.
Useful7/10
Difficulty5/10
Novelty5/10