# Hebbian SDR Adapter for Streaming Context Learning

- ID: 2621
- Canonical URL: https://synthcore.org/idea/2621/hebbian-sdr-adapter-for-streaming-context-learning
- API JSON: https://synthcore.org/api/idea/2621.json
- API Markdown: https://synthcore.org/api/idea/2621.md
- Verification status: unverified
- Source: [arXiv:2608.20408](https://arxiv.org/abs/2608.20408)
- Category: training
- Solves: sample-efficiency, stability, memory
- ML areas: graph-nn, fine-tuning, embedding
- Math tags: probability, statistics, dynamical-systems
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Attach a local BCM-trained binary adapter to a pretrained or frozen encoder, allowing new graph nodes or streaming examples to acquire representations without backpropagating through the main network. The adapter learns only from positive co-occurrence statistics and maintains sparse codes, providing a low-memory continual-learning path.

## Mathematical statement

Let h in R^d be a frozen encoder output and c a context identifier. Maintain online co-occurrence counts C_ic between a hashed feature i of h and context c, together with marginals C_i_dot and C_dot_c. Compute q_ic=max(0, log(p(i,c)/(p(i)p(c)))) using smoothed empirical probabilities. For adapter column j, response y_j=ReLU(w_j^T h), threshold theta_j is an exponential estimate of y_j^2, and the local update is Delta w_j=eta q_ic h y_j(y_j-theta_j). After updating, emit z in {0,1}^m by retaining the k largest responses. No downstream gradient is required for the adapter. Frequently overactive columns have y_j below their running threshold and are depressed, while selectively responding columns with y_j above threshold are potentiated.

## Key formulas

- $$q_{ic}=\max\left\{0,\log\frac{p(i,c)}{p(i)p(c)}\right\},\qquad p(i,c)=\frac{C_{ic}+\delta}{N+\delta\lvert\mathcal{I}\rvert\lvert\mathcal{C}\rvert}.$$
- $$y_j=\operatorname{ReLU}(w_j^\top h),\qquad w_j\leftarrow w_j+\eta q_{ic}h\,y_j(y_j-\theta_j),\qquad \theta_j\leftarrow(1-\alpha)\theta_j+\alpha y_j^2.$$
- $$z_j=\mathbf{1}\left[j\in\operatorname{TopK}(y,k)\right],\qquad \operatorname{storage}(z)=O(k\log m)\ \text{bits rather than }O(m)\ \text{floating-point values}.$$

## Implementation notes

Place the adapter after a frozen encoder: for a graph model use a pretrained node feature encoder h=f(x_u), while for a language or multimodal stream use the frozen item representation h. Hash or randomly project h to dimension d if necessary. For every streaming example, identify a context c from a neighboring graph node, a co-occurring token, a label-free augmentation, or a random-walk window. Update a count sketch for feature-context pairs, compute q from smoothed counts, and perform the BCM update on the adapter columns. To reduce cost, update only the top L response columns plus a small random subset of non-top columns; compute the final k-winner code from all responses or a candidate index. Clip the norm of h and renormalize w_j after each minibatch. Keep the base encoder frozen and train only a linear classifier on SDRs in the first experiment, then compare against full fine-tuning and a replay-based adapter. Use a two-layer GraphSAGE encoder on Cora or ogbn-arxiv, expose new nodes in temporal batches, and evaluate accuracy after every batch, trainable parameters, peak memory, and forgetting on earlier nodes. The first success criterion is at least ten times fewer trainable parameters than full fine-tuning, final accuracy within two percentage points, and a smaller accuracy drop when the stream changes and old examples are not replayed.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.