# Fading-Memory Habituation Gate

- ID: 1625
- Canonical URL: https://synthcore.org/idea/1625/fading-memory-habituation-gate
- API JSON: https://synthcore.org/api/idea/1625.json
- API Markdown: https://synthcore.org/api/idea/1625.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.00249](https://arxiv.org/abs/2608.00249)
- Category: dynamics
- Solves: stability, generalization, inference-speedup
- ML areas: attention, transformer, rnn, inference-speedup
- Math tags: dynamical-systems, control-theory, linear-algebra, stochastic-processes
- Ratings: usefulness 7/10; difficulty 4/10; novelty 6/10

## Idea description

Add a per-feature or per-token state that accumulates recent stimulation and decays when stimulation is absent, then use a nonlinear decreasing gain to suppress repeatedly activated features. This creates short-term adaptation without changing the core transformer or recurrent weights: familiar inputs are processed with reduced gain, while novel inputs recover their full response.

## Mathematical statement

For feature stimulation magnitude $s_t\ge 0$, maintain a fading-memory state $a_t\ge 0$ with decay factor $\rho\in(0,1)$ and update $a_{t+1}=\rho a_t+(1-\rho)s_t$. Let the neural preactivation or value vector be $v_t\in\mathbb{R}^d$, and let $s_t$ be either $|v_t|$ componentwise or a scalar token novelty score. Apply the nonlinear habituation gain $g(a_t)=1/(1+\beta a_t)$, with $\beta\ge 0$, to obtain $\tilde v_t=g(a_t)\odot v_t$; for scalar $a_t$, the same gain multiplies every component. The fading-memory pole $\rho$ gives recovery time $\tau=-1/\log\rho$ timesteps. Under constant stimulation $s_t=s>0$, the state converges to $a_\infty=s$, so the asymptotic response gain is $g_\infty=1/(1+\beta s)<1$; after withdrawal, $a_{t+k}=\rho^k a_t$ and the gain recovers exponentially toward one. The paper's structural claim is that an LTI input-output map cannot produce both persistent response decrement to repeated stimulation and recovery after withdrawal; the static nonlinearity composed with fading memory supplies the missing mechanism.

## Key formulas

- $$a_{t+1}=\rho a_t+(1-\rho)s_t,\qquad 0<\rho<1,$$
- $$\tilde v_t=g(a_t)\odot v_t,\qquad g(a)=\frac{1}{1+\beta a},\quad \beta\ge 0,$$
- $$a_{t+k}=\rho^k a_t\ \text{after withdrawal},\qquad g(a_{t+k})=\frac{1}{1+\beta\rho^k a_t},\qquad \tau=-\frac{1}{\log\rho},$$
- $$\mathcal{C}=\{\Sigma:(u,y)\in\mathcal{B}(\Sigma)\text{ satisfies }H_1,H_2,\ldots\text{ for all admissible }u\}.$$

## Implementation notes

1. Integration point: start with a transformer attention block. For each token position and feature channel, compute the usual value or MLP vector $v_t$; define stimulation $s_t=\operatorname{stopgrad}(\|v_t\|_2/\sqrt d)$ for a scalar gate per token, or $s_t=\operatorname{stopgrad}(|v_t|)$ for a channelwise gate. Store $a_t$ in the KV-cache during autoregressive inference, or reset it at the beginning of every training sequence. Replace $v_t$ by $\tilde v_t=v_t/(1+\beta a_t)$. 2. Pseudocode: initialize $a=0$; at each timestep compute $v$ and $s$; update $a\leftarrow\rho a+(1-\rho)s$; compute $g\leftarrow1/(1+\beta a)$; pass $g\odot v$ to attention or the MLP residual stream. Parameterize $\rho=\operatorname{sigmoid}(r)$ and $\beta=\operatorname{softplus}(b)$, but initially sweep fixed values. 3. Computed from the mechanism: the exponential state update, nonlinear gain, and predicted recovery time $\tau$. Estimated empirically: the effective response amplitude, fitted decay factor, and whether adaptation harms novel-token accuracy; use stop-gradient for $s$ in the first experiment to isolate dynamics. 4. First cheap experiment: train a small causal transformer on repeated-copy sequences mixed with randomly sampled sequences, comparing the baseline, an LTI-only gate $\tilde v_t=(1-ca_t)v_t$, and the nonlinear gate. At inference, present $m$ identical pulses followed by $k$ blank or novel-token steps. The predicted signature is a monotone response decrease toward $1/(1+\beta s)$ during repetition and exponential recovery with fitted slope $\log\rho$; the measured half-recovery time should be $k_{1/2}=\log(1/2)/\log\rho$ within 20%. Sweep $\rho$ and verify that stability is retained for all $\rho<1$, while setting $\rho\ge1$ produces nondecaying or divergent adaptation state. Also test the paper's structural prediction: the matched LTI gate should fail to show both bounded repeated suppression and recovery toward the original response unless an additional nonlinear saturation is introduced.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built and ran a reproducible fading-memory habituation gate verification. The three quantitative predictions were confirmed: steady-state gain matched 1/(1+beta*s) with maximum absolute error 2.0e-14, recovery half-indices exactly matched ceil(log(0.5)/log(rho)) for all tested rho, and fitted decay slopes matched log(rho) to below 2e-16. The pole sweep showed decay for rho<1, constant state at rho=1, and growth for rho>1; the controlled comparison showed repeated-pulse gain 0.363 versus baseline 1.0, with recovery to 0.824 after withdrawal, so the mechanism clearly manifested, though no trained-transformer or task-accuracy win was established.

### Mechanism check

- Verdict: Built and ran a reproducible fading-memory habituation gate verification. The three quantitative predictions were confirmed: steady-state gain matched 1/(1+beta*s) with maximum absolute error 2.0e-14, recovery half-indices exactly matched ceil(log(0.5)/log(rho)) for all tested rho, and fitted decay slopes matched log(rho) to below 2e-16. The pole sweep showed decay for rho<1, constant state at rho=1, and growth for rho>1; the controlled comparison showed repeated-pulse gain 0.363 versus baseline 1.0, with recovery to 0.824 after withdrawal, so the mechanism clearly manifested, though no trained-transformer or task-accuracy win was established.
- Confidence: 9/10
- Limitations: Only analytical/numerical toy dynamics and a controlled pulse-sequence comparison were tested. No causal transformer training, language-modeling accuracy, inference latency, KV-cache integration, or robustness/generalization evaluation was performed.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_report.json](https://synthcore.org/code/606/bench_report.json)
- [habituation_experiment.py](https://synthcore.org/code/606/habituation_experiment.py)
- [report.md](https://synthcore.org/code/606/report.md)
- [report_bench_2026-09-02T225317.md](https://synthcore.org/code/606/report_bench_2026-09-02T225317.md)
- [results.json](https://synthcore.org/code/606/results.json)
- [stage2_bench.py](https://synthcore.org/code/606/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/606)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.