{
 "artifacts": null,
 "category": "memory",
 "description": "Add a recurrent associative matrix to each selected transformer layer so recent key-value relationships can be retrieved without retaining every past token or performing gradient updates. The matrix uses input-dependent retention and write gates, but retrieval is always performed from the pre-write state, preventing the current target from leaking into its own prediction. Frobenius-norm clipping makes the recurrent memory bounded and provides a direct stability control.",
 "formulas_latex": [
  "$$Q_t=X_tW_Q,\\qquad K_t=X_tW_K,\\qquad V_t=X_tW_V$$",
  "$$A_t=\\operatorname{clip}\\left(\\frac{K_t^{\\top}V_t}{\\sqrt{T}}\\right)$$",
  "$$M_t=\\operatorname{clipnorm}\\left(\\lambda_tM_{t-1}+\\eta_tA_t,c_M\\right)$$",
  "$$Y_t^{\\mathrm{mem}}=Q_tM_{t-1},\\qquad \\lambda_t=\\sigma(g_\\lambda(\\operatorname{pool}(X_t))),\\qquad \\eta_t=\\operatorname{softplus}(g_\\eta(\\operatorname{pool}(X_t)))$$"
 ],
 "id": 2970,
 "implementation": "1. Integration point: add one memory branch to a transformer self-attention block, preferably after the Q/K/V projections and before the output projection. Maintain one matrix M per attention head or per layer, with shape d_k by d_v; detach it from autograd between examples or episodes so it behaves as state rather than an ever-growing computation graph. Compute standard self-attention as usual, and compute the auxiliary memory values Y_mem=Q M_old. Add this branch to the normal attention output with a learned scalar mixing gate.\n\n2. Pseudocode:\n```text\nstate M_old = zeros(d_k, d_v)\nfor each sequence X:\n    Q = X @ WQ; K = X @ WK; V = X @ WV\n    h = mean(X, axis=tokens)\n    lambda = sigmoid(controller_lambda(h))\n    eta = softplus(controller_eta(h))\n    Y_mem = Q @ stop_gradient(M_old)\n    Y_attn = softmax(Q @ K.T / sqrt(d_k)) @ V\n    Y = Y_attn + sigmoid(controller_mix(h)) * Y_mem\n    A = clip((K.T @ V) / sqrt(T), -a_max, a_max)\n    Z = lambda * M_old + eta * A\n    M_new = Z * min(1, c_M / (frobenius_norm(Z) + 1e-6))\n    emit prediction from Y\n    M_old = stop_gradient(M_new)\n```\nThe displayed equations define A, the recurrence, and the bounded read/write mechanism. The controller functions g_lambda and g_eta can be small learned linear layers or a two-layer MLP. Estimate no mathematical constants: c_M, a_max, and the initial memory mixing coefficient are hyperparameters. Log ||M||_F, average lambda, average eta, and the ratio of memory-branch to attention-branch norms to detect saturation or collapse.\n\n3. First experiment: implement this in a 2-4 layer ViT-Tiny or small causal transformer on sequential Omniglot 5-way 1-shot episodes and a CORe50-style class-incremental stream. Compare against the same backbone with no memory, a fixed-retention fast-weight recurrence with constant lambda and eta, and a replay buffer or recurrent-attention baseline with matched parameter and memory budgets. Train the controller and projection layers initially, then jointly train all parameters. The primary success signal is higher delayed-association accuracy after distractor episodes at equal stored-state bytes; secondary signals are lower forgetting, stable bounded ||M||_F, and faster adaptation than the no-memory baseline. An ablation removing read-before-write should be tested for artificial same-step gains or worse held-out temporal accuracy.",
 "math_summary": "The paper forms projected keys and values from the layer input X_t: Q_t=X_tW_Q, K_t=X_tW_K, and V_t=X_tW_V, where X_t is in R^{T x d}, T is the number of tokens, W_Q, W_K, and W_V are learned projection matrices, and d is the projection width. The key-value coactivation matrix is A_t=clip(K_t^T V_t/sqrt(T)), where A_t is in R^{d_k x d_v}, d_k and d_v are key and value widths, and clip is elementwise clipping to a fixed interval. The memory recurrence is M_t=clipnorm(lambda_t M_{t-1}+eta_t A_t,c_M), where M_{t-1} is the state available before the current write, lambda_t in [0,1] is retention, eta_t is nonnegative write strength, and c_M is the maximum Frobenius norm. Define clipnorm(Z,c)=Z min(1,c/(||Z||_F+epsilon)). The adaptation reads the old state with Y_t^mem=Q_t M_{t-1} and writes only after producing the layer output. This creates a bounded linear dynamical system when lambda_t is below one, while eta_t controls rapid association formation.",
 "math_tags": [
  "linear-algebra",
  "dynamical-systems",
  "optimization"
 ],
 "ml_areas": [
  "transformer",
  "attention",
  "training-dynamics",
  "fine-tuning"
 ],
 "paper": {
  "arxiv_id": "2609.00358",
  "arxiv_url": "https://arxiv.org/abs/2609.00358",
  "summary_what_math_gives_to_ml": "The paper provides a concrete fast-weight associative memory update that separates retrieval from plasticity: the model reads a pre-update memory state and only then writes the current key-value association. The transferable asset is a bounded, exponentially retained Hebbian matrix whose retention and write strength can vary per input, giving a cheap alternative to storing all past tokens or updating the full transformer. A practical adaptation is to insert this memory into an attention block as an auxiliary value source, with controller outputs determining input-dependent retention and plasticity. The main risk is that the underlying Hebbian rule is related to existing fast-weight transformers, so its benefit must be demonstrated against recurrent attention and explicit replay baselines.",
  "title": "Where Should Experience Live? Hierarchical Hebbian Memory for Continual Vision Transformers",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 4,
  "usefulness": 6
 },
 "solves": [
  "memory",
  "sample-efficiency",
  "stability"
 ],
 "title": "Bounded Adaptive Hebbian Fast-Weight Cache",
 "url": "https://synthcore.org/idea/2970/bounded-adaptive-hebbian-fast-weight-cache",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}