Bounded Adaptive Hebbian Fast-Weight Cache

1. Integration point: add one memory branch to a transformer self-attention block, preferably after the Q/K/V projections and before the output projection. Maintain one matrix M per attention head or per layer, with shape d_k by d_v; detach it from autograd between examples or episodes so it behaves as state rather than an ever-growing computation graph. Compute standard self-attention as usual, and compute the auxiliary memory values Y_mem=Q M_old. Add this branch to the normal attention output with a learned scalar mixing gate.

2. Pseudocode:

state M_old = zeros(d_k, d_v)
for each sequence X:
    Q = X @ WQ; K = X @ WK; V = X @ WV
    h = mean(X, axis=tokens)
    lambda = sigmoid(controller_lambda(h))
    eta = softplus(controller_eta(h))
    Y_mem = Q @ stop_gradient(M_old)
    Y_attn = softmax(Q @ K.T / sqrt(d_k)) @ V
    Y = Y_attn + sigmoid(controller_mix(h)) * Y_mem
    A = clip((K.T @ V) / sqrt(T), -a_max, a_max)
    Z = lambda * M_old + eta * A
    M_new = Z * min(1, c_M / (frobenius_norm(Z) + 1e-6))
    emit prediction from Y
    M_old = stop_gradient(M_new)

The displayed equations define A, the recurrence, and the bounded read/write mechanism. The controller functions g_lambda and g_eta can be small learned linear layers or a two-layer MLP. Estimate no mathematical constants: c_M, a_max, and the initial memory mixing coefficient are hyperparameters. Log ||M||_F, average lambda, average eta, and the ratio of memory-branch to attention-branch norms to detect saturation or collapse.

3. First experiment: implement this in a 2-4 layer ViT-Tiny or small causal transformer on sequential Omniglot 5-way 1-shot episodes and a CORe50-style class-incremental stream. Compare against the same backbone with no memory, a fixed-retention fast-weight recurrence with constant lambda and eta, and a replay buffer or recurrent-attention baseline with matched parameter and memory budgets. Train the controller and projection layers initially, then jointly train all parameters. The primary success signal is higher delayed-association accuracy after distractor episodes at equal stored-state bytes; secondary signals are lower forgetting, stable bounded ||M||_F, and faster adaptation than the no-memory baseline. An ablation removing read-before-write should be tested for artificial same-step gains or worse held-out temporal accuracy.