△ Mechanism confirmed, baseline not beaten 2026

Fading-Memory Habituation Gate

Implementation & benchmark of arXiv:2608.00249 — Dynamical principles of habituation across substrates and scales

Usefulness7/10

Difficulty4/10

Novelty6/10

Dynamics Stability Generalization Inference speedup Attention Transformer Rnn Inference speedup Dynamical systems Control theory Linear algebra Stochastic processes

Source paper: Dynamical principles of habituation across substrates and scales arXiv:2608.00249 ⓘ · analyzed Aug 31, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Add a per-feature or per-token state that accumulates recent stimulation and decays when stimulation is absent, then use a nonlinear decreasing gain to suppress repeatedly activated features. This creates short-term adaptation without changing the core transformer or recurrent weights: familiar inputs are processed with reduced gain, while novel inputs recover their full response.

Formulas

$$a_{t+1}=\rho a_t+(1-\rho)s_t,\qquad 0<\rho<1,$$

$$\tilde v_t=g(a_t)\odot v_t,\qquad g(a)=\frac{1}{1+\beta a},\quad \beta\ge 0,$$

$$a_{t+k}=\rho^k a_t\ \text{after withdrawal},\qquad g(a_{t+k})=\frac{1}{1+\beta\rho^k a_t},\qquad \tau=-\frac{1}{\log\rho},$$

$$\mathcal{C}=\{\Sigma:(u,y)\in\mathcal{B}(\Sigma)\text{ satisfies }H_1,H_2,\ldots\text{ for all admissible }u\}.$$

Mathematical statement

For feature stimulation magnitude $s_t\ge 0$, maintain a fading-memory state $a_t\ge 0$ with decay factor $\rho\in(0,1)$ and update $a_{t+1}=\rho a_t+(1-\rho)s_t$. Let the neural preactivation or value vector be $v_t\in\mathbb{R}^d$, and let $s_t$ be either $|v_t|$ componentwise or a scalar token novelty score. Apply the nonlinear habituation gain $g(a_t)=1/(1+\beta a_t)$, with $\beta\ge 0$, to obtain $\tilde v_t=g(a_t)\odot v_t$; for scalar $a_t$, the same gain multiplies every component. The fading-memory pole $\rho$ gives recovery time $\tau=-1/\log\rho$ timesteps. Under constant stimulation $s_t=s>0$, the state converges to $a_\infty=s$, so the asymptotic response gain is $g_\infty=1/(1+\beta s)<1$; after withdrawal, $a_{t+k}=\rho^k a_t$ and the gain recovers exponentially toward one. The paper's structural claim is that an LTI input-output map cannot produce both persistent response decrement to repeated stimulation and recovery after withdrawal; the static nonlinearity composed with fading memory supplies the missing mechanism.

Implementation notes

1. Integration point: start with a transformer attention block. For each token position and feature channel, compute the usual value or MLP vector $v_t$; define stimulation $s_t=\operatorname{stopgrad}(\|v_t\|_2/\sqrt d)$ for a scalar gate per token, or $s_t=\operatorname{stopgrad}(|v_t|)$ for a channelwise gate. Store $a_t$ in the KV-cache during autoregressive inference, or reset it at the beginning of every training sequence. Replace $v_t$ by $\tilde v_t=v_t/(1+\beta a_t)$. 2. Pseudocode: initialize $a=0$; at each timestep compute $v$ and $s$; update $a\leftarrow\rho a+(1-\rho)s$; compute $g\leftarrow1/(1+\beta a)$; pass $g\odot v$ to attention or the MLP residual stream. Parameterize $\rho=\operatorname{sigmoid}(r)$ and $\beta=\operatorname{softplus}(b)$, but initially sweep fixed values. 3. Computed from the mechanism: the exponential state update, nonlinear gain, and predicted recovery time $\tau$. Estimated empirically: the effective response amplitude, fitted decay factor, and whether adaptation harms novel-token accuracy; use stop-gradient for $s$ in the first experiment to isolate dynamics. 4. First cheap experiment: train a small causal transformer on repeated-copy sequences mixed with randomly sampled sequences, comparing the baseline, an LTI-only gate $\tilde v_t=(1-ca_t)v_t$, and the nonlinear gate. At inference, present $m$ identical pulses followed by $k$ blank or novel-token steps. The predicted signature is a monotone response decrease toward $1/(1+\beta s)$ during repetition and exponential recovery with fitted slope $\log\rho$; the measured half-recovery time should be $k_{1/2}=\log(1/2)/\log\rho$ within 20%. Sweep $\rho$ and verify that stability is retained for all $\rho<1$, while setting $\rho\ge1$ produces nondecaying or divergent adaptation state. Also test the paper's structural prediction: the matched LTI gate should fail to show both bounded repeated suppression and recovery toward the original response unless an additional nonlinear saturation is introduced.

Verification

Mechanism confirmed, baseline not beaten

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built and ran a reproducible fading-memory habituation gate verification. The three quantitative predictions were confirmed: steady-state gain matched 1/(1+beta*s) with maximum absolute error 2.0e-14, recovery half-indices exactly matched ceil(log(0.5)/log(rho)) for all tested rho, and fitted decay slopes matched log(rho) to below 2e-16. The pole sweep showed decay for rho<1, constant state at rho=1, and growth for rho>1; the controlled comparison showed repeated-pulse gain 0.363 versus baseline 1.0, with recovery to 0.824 after withdrawal, so the mechanism clearly manifested, though no trained-transformer or task-accuracy win was established.

Agent confidence: 9/10

Baseline: Unmodified gain: repeated pulse 1.000, post-withdrawal 1.000, novel pulse 1.000.
Idea: Nonlinear gate (rho=0.9, beta=2): repeated pulse gain 0.3627, post-withdrawal gain 0.8240, novel pulse after reset 0.8333. Matched linear state gate: repeated 0.4144 and post-withdrawal 0.9288.

Limitations: Only analytical/numerical toy dynamics and a controlled pulse-sequence comparison were tested. No causal transformer training, language-modeling accuracy, inference latency, KV-cache integration, or robustness/generalization evaluation was performed.

How to run: python3 habituation_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.00249) Sep 2, 2026

Significantly WORSE than the tuned baseline (+3.9%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Sequence modeling
Model: transformer_tiny
Paired seeds: 8
Baseline mean: 0.1342
Idea mean: 0.1394
Effect (Δ): 0.0052 (+3.9%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Confirmed ✓
Practical effect: Hurts
Baseline tuning: swept over 3 configs

Limitations:

Only the structurally matched sequence forecast track was tested; no inference KV-cache implementation, latency measurement, other bench tracks, or broader rho/beta sweep was evaluated.

How to run: python3 stage2_bench.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Sep 2, 2026 latest Failed ✗

Agent confidence: 10/10

Implemented the fading-memory habituation gate in a matched sequence transformer_tiny system and ran the full 8-seed paired protocol with a parity-preserving baseline learning-rate sweep. The mechanism matched its predicted steady-state gain and recovery slope, but trained task performance was significantly worse: idea test MSE 0.13939 versus baseline 0.13418, delta +0.00520, permutation p=0.0081, with 0/8 paired wins. Thus the mechanism transfers mathematically but not as a benchmark performance win.

Baseline: sequence/transformer_tiny; best lr=0.0015; full 8-seed mean test MSE 0.1341835428, std 0.0076598566
Comparison: delta_mean=+0.0052043395, p_value=0.0081, idea_wins=0/8, verdict=idea worse (significant)
Idea: best cfg lr=0.0015, rho=0.9, beta=0.5; full 8-seed mean test MSE 0.1393878823, std 0.0115220628
Mechanism signature: trained model mean_state=0.6669986, mean_gain=0.7588040, first_gain=0.9565723, last_gain=0.7013749; predicted-vs-observed steady gain absolute error=1.67e-16 and recovery slope absolute error=0.0; confirmed=true

Benchmark result