Memory ideas

External memories, state compression, retrieval-style read/write policies and calibrated stopping rules.

Unverified 2026

Bounded Adaptive Hebbian Fast-Weight Cache

Add a recurrent associative matrix to each selected transformer layer so recent key-value relationships can be retrieved without retaining every past token or performing gradient updates. The matrix uses input-dependent retention and write gates, but retrieval is always performed from the pre-write state, preventing the current target from leaking into its own prediction. Frobenius-norm clipping makes the recurrent memory bounded and provides a direct stability control.

Useful6/10
Difficulty4/10
Novelty4/10
Paper: Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers arXiv:2609.00358