# Dendritic LMS State for Spiking ICL

- ID: 359
- Canonical URL: https://synthcore.org/idea/359/dendritic-lms-state-for-spiking-icl
- API JSON: https://synthcore.org/api/idea/359.json
- API Markdown: https://synthcore.org/api/idea/359.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.02283](https://arxiv.org/abs/2607.02283)
- Category: architecture
- Solves: accuracy, stability, memory
- ML areas: rnn, ssm, architecture, training
- Math tags: optimization, dynamical-systems, linear-algebra
- Ratings: usefulness 8/10; difficulty 5/10; novelty 7/10

## Idea description

Give a single spiking layer a persistent vector-valued apical compartment that stores the current online linear predictor for the task. On each labeled context pair, its subthreshold state performs a leaky LMS update; on the query, the state is read without updating, allowing in-context adaptation without attention or inference-time synaptic plasticity.

## Mathematical statement

The paper identifies the apical recurrence with leaky online Widrow–Hoff LMS. Let $x_t\in\mathbb{R}^d$ be the feature vector at context step $t$, $y_t\in\mathbb{R}$ its target, $w_t\in\mathbb{R}^d$ the persistent dendritic state representing the current task predictor, $\hat y_t=w_t^\top x_t$ the prediction, $e_t=y_t-\hat y_t$ the prediction error, $\eta>0$ the learning rate, and $\lambda\in[0,1]$ the leak or forgetting factor. The online LMS update is $w_{t+1}=\lambda w_t+\eta e_t x_t$. Equivalently, it is gradient descent on the instantaneous squared loss $\ell_t(w)=\tfrac12(y_t-w^\top x_t)^2$, since $\nabla_w\ell_t(w_t)=-e_t x_t$. The dendritic implementation uses the subthreshold voltage $a_t$ as $w_t$; a separate soma converts $a_t$ into spikes, while the apical state remains persistent across examples. For vector targets $y_t\in\mathbb{R}^k$, replace $w_t$ by $W_t\in\mathbb{R}^{k\times d}$ and use $W_{t+1}=\lambda W_t+\eta(y_t-W_tx_t)x_t^\top$. The useful property is that adaptation is a forward-pass state transition with $O(d)$ state and $O(d)$ update cost per example, rather than a learned deep computation or parameter mutation.

## Key formulas

- $$\hat y_t = w_t^\top x_t, \qquad e_t = y_t-\hat y_t,$$
- $$w_{t+1}=\lambda w_t+\eta e_t x_t,$$
- $$w_{t+1}=\lambda w_t-\eta\nabla_w\left[\frac12\left(y_t-w_t^\top x_t\right)^2\right],$$
- $$\hat y_{\mathrm{query}}=w_T^\top x_{\mathrm{query}}, \qquad \text{with no update after the query}.$$

## Implementation notes

Integrate this at the hidden-state level of a small recurrent SNN or a single-layer spiking token processor. Encode each scalar or vector input example into a feature vector $x_t\in\mathbb{R}^d$ using a fixed random projection or a learned linear encoder. Maintain one persistent floating-point apical tensor $a_t\in\mathbb{R}^d$ per sequence and initialize it to zero at the beginning of every task. For a labeled context pair $(x_t,y_t)$, compute the prediction $\hat y_t=a_t^\top x_t$, the error $e_t=y_t-\hat y_t$, and update the apical state with $a\leftarrow\lambda a+\eta e_t x_t$. Feed $a$ plus the current input through a learned soma/readout layer, apply a spiking nonlinearity such as an LIF threshold, and use surrogate gradients only for the encoder, readout, and optional spike parameters. On the query token, compute $\hat y=a^\top x_{query}$ and freeze $a$; do not accidentally train or update it from the query loss during the forward pass. A compact pseudocode loop is: `a=zeros(d); for (x,y,query_flag) in sequence: z=encoder(x); pred=a@z; if not query_flag: err=y-pred; a=lambda*a+eta*err*z; spikes=surrogate_soma(a,z); output=readout(spikes,pred)`. The mathematical quantities $\lambda$, $\eta$, the LMS error, and the outer-product update are computed exactly; feature normalization, spike thresholds, and the encoder are learned or estimated empirically. Start with the scalar linear Garg-2022 protocol at $d=20,50,100$, compare against a Pure LIF network and a similarly sized GRU or Transformer, and run at least 20 random seeds. Success means higher query $R^2$, substantially lower seed variance, and performance that remains stable as task dimension exceeds the baseline's grokking or collapse regime. Also record state size, synaptic operations, and wall-clock time to test whether the single-layer state gives an inference-memory or latency advantage.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a reproducible dendritic LMS state MVP with exact query-time freezing and a minimal LIF soma helper. The numerical gradient/update checks passed, with maximum errors of 2.2e-9 and 4.4e-16, and the scalar recurrence was contractive for the tested parameters. Leaky LMS improved over zero adaptation but did not consistently beat standard non-leaky LMS: MSE was 0.03542 vs 0.03535 at d=20, 0.01760 vs 0.01764 at d=50, and 0.00958 vs 0.00963 at d=100; this is not a clear effect attributable to the proposed leak.

### Mechanism check

- Verdict: Built a reproducible dendritic LMS state MVP with exact query-time freezing and a minimal LIF soma helper. The numerical gradient/update checks passed, with maximum errors of 2.2e-9 and 4.4e-16, and the scalar recurrence was contractive for the tested parameters. Leaky LMS improved over zero adaptation but did not consistently beat standard non-leaky LMS: MSE was 0.03542 vs 0.03535 at d=20, 0.01760 vs 0.01764 at d=50, and 0.00958 vs 0.00963 at d=100; this is not a clear effect attributable to the proposed leak.
- Confidence: 9/10
- Limitations: The experiment used synthetic unit-normalized linear regression rather than a trained end-to-end spiking network, and lambda/eta were fixed rather than tuned per method. It did not measure GPU latency, FLOPs, memory, learned encoders, spike-based optimization, or compare against GRU/Transformer baselines; the LIF helper was not part of the benchmarked predictor.

## Artifacts

- [experiment.py](https://synthcore.org/code/54/experiment.py)
- [report.md](https://synthcore.org/code/54/report.md)
- [results.json](https://synthcore.org/code/54/results.json)
- [Download all files as ZIP](https://synthcore.org/download/54)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.