Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers

What the math gives to ML

The paper provides a concrete fast-weight associative memory update that separates retrieval from plasticity: the model reads a pre-update memory state and only then writes the current key-value association. The transferable asset is a bounded, exponentially retained Hebbian matrix whose retention and write strength can vary per input, giving a cheap alternative to storing all past tokens or updating the full transformer. A practical adaptation is to insert this memory into an attention block as an auxiliary value source, with controller outputs determining input-dependent retention and plasticity. The main risk is that the underlying Hebbian rule is related to existing fast-weight transformers, so its benefit must be demonstrated against recurrent attention and explicit replay baselines.