# Bounded Adaptive Hebbian Fast-Weight Cache

- ID: 2970
- Canonical URL: https://synthcore.org/idea/2970/bounded-adaptive-hebbian-fast-weight-cache
- API JSON: https://synthcore.org/api/idea/2970.json
- API Markdown: https://synthcore.org/api/idea/2970.md
- Verification status: unverified
- Source: [arXiv:2609.00358](https://arxiv.org/abs/2609.00358)
- Category: memory
- Solves: memory, sample-efficiency, stability
- ML areas: transformer, attention, training-dynamics, fine-tuning
- Math tags: linear-algebra, dynamical-systems, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 4/10

## Idea description

Add a recurrent associative matrix to each selected transformer layer so recent key-value relationships can be retrieved without retaining every past token or performing gradient updates. The matrix uses input-dependent retention and write gates, but retrieval is always performed from the pre-write state, preventing the current target from leaking into its own prediction. Frobenius-norm clipping makes the recurrent memory bounded and provides a direct stability control.

## Mathematical statement

The paper forms projected keys and values from the layer input X_t: Q_t=X_tW_Q, K_t=X_tW_K, and V_t=X_tW_V, where X_t is in R^{T x d}, T is the number of tokens, W_Q, W_K, and W_V are learned projection matrices, and d is the projection width. The key-value coactivation matrix is A_t=clip(K_t^T V_t/sqrt(T)), where A_t is in R^{d_k x d_v}, d_k and d_v are key and value widths, and clip is elementwise clipping to a fixed interval. The memory recurrence is M_t=clipnorm(lambda_t M_{t-1}+eta_t A_t,c_M), where M_{t-1} is the state available before the current write, lambda_t in [0,1] is retention, eta_t is nonnegative write strength, and c_M is the maximum Frobenius norm. Define clipnorm(Z,c)=Z min(1,c/(||Z||_F+epsilon)). The adaptation reads the old state with Y_t^mem=Q_t M_{t-1} and writes only after producing the layer output. This creates a bounded linear dynamical system when lambda_t is below one, while eta_t controls rapid association formation.

## Key formulas

- $$Q_t=X_tW_Q,\qquad K_t=X_tW_K,\qquad V_t=X_tW_V$$
- $$A_t=\operatorname{clip}\left(\frac{K_t^{\top}V_t}{\sqrt{T}}\right)$$
- $$M_t=\operatorname{clipnorm}\left(\lambda_tM_{t-1}+\eta_tA_t,c_M\right)$$
- $$Y_t^{\mathrm{mem}}=Q_tM_{t-1},\qquad \lambda_t=\sigma(g_\lambda(\operatorname{pool}(X_t))),\qquad \eta_t=\operatorname{softplus}(g_\eta(\operatorname{pool}(X_t)))$$

## Implementation notes

1. Integration point: add one memory branch to a transformer self-attention block, preferably after the Q/K/V projections and before the output projection. Maintain one matrix M per attention head or per layer, with shape d_k by d_v; detach it from autograd between examples or episodes so it behaves as state rather than an ever-growing computation graph. Compute standard self-attention as usual, and compute the auxiliary memory values Y_mem=Q M_old. Add this branch to the normal attention output with a learned scalar mixing gate.

2. Pseudocode:
```text
state M_old = zeros(d_k, d_v)
for each sequence X:
    Q = X @ WQ; K = X @ WK; V = X @ WV
    h = mean(X, axis=tokens)
    lambda = sigmoid(controller_lambda(h))
    eta = softplus(controller_eta(h))
    Y_mem = Q @ stop_gradient(M_old)
    Y_attn = softmax(Q @ K.T / sqrt(d_k)) @ V
    Y = Y_attn + sigmoid(controller_mix(h)) * Y_mem
    A = clip((K.T @ V) / sqrt(T), -a_max, a_max)
    Z = lambda * M_old + eta * A
    M_new = Z * min(1, c_M / (frobenius_norm(Z) + 1e-6))
    emit prediction from Y
    M_old = stop_gradient(M_new)
```
The displayed equations define A, the recurrence, and the bounded read/write mechanism. The controller functions g_lambda and g_eta can be small learned linear layers or a two-layer MLP. Estimate no mathematical constants: c_M, a_max, and the initial memory mixing coefficient are hyperparameters. Log ||M||_F, average lambda, average eta, and the ratio of memory-branch to attention-branch norms to detect saturation or collapse.

3. First experiment: implement this in a 2-4 layer ViT-Tiny or small causal transformer on sequential Omniglot 5-way 1-shot episodes and a CORe50-style class-incremental stream. Compare against the same backbone with no memory, a fixed-retention fast-weight recurrence with constant lambda and eta, and a replay buffer or recurrent-attention baseline with matched parameter and memory budgets. Train the controller and projection layers initially, then jointly train all parameters. The primary success signal is higher delayed-association accuracy after distractor episodes at equal stored-state bytes; secondary signals are lower forgetting, stable bounded ||M||_F, and faster adaptation than the no-memory baseline. An ablation removing read-before-write should be tested for artificial same-step gains or worse held-out temporal accuracy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.