DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule

What the math gives to ML

The paper identifies a structural mismatch in conventional momentum: for a linear layer, each per-example gradient is a key-value outer product, but EMA treats all input directions as if they had identical visitation frequency. Its transferable asset is the delta-rule interpretation of the momentum matrix as an associative memory, allowing the update to correct only the key direction currently being queried rather than globally decaying every direction. This suggests a drop-in optimizer update for linear, convolutional, attention-projection, and MLP weights that tracks drifting gradients more accurately without storing additional history. The most direct test is to replace only the momentum-buffer update in SGD or AdamW and compare loss-per-step, stale-direction recovery, and wall-clock overhead against EMA momentum.