Idea description

Insert a slow routing state and an intermediate hysteresis variable between a neural memory and its next-state selector. The hysteresis prevents small prediction fluctuations from repeatedly changing the active attractor, while the slower router learns transition probabilities independently of the attractor parameters.

Formulas

$$\tau_x\dot x=f_a(x),\qquad \tau_h\dot h=\sigma(g(r,x))-h,\qquad \tau_r\dot r=F(r,u),\qquad \tau_x\ll\tau_h\ll\tau_r.$$

$$a\to b\ \text{only if}\ h_b\geq\eta_{\mathrm{on}},\qquad b\to a\ \text{is forbidden until}\ h_b\leq\eta_{\mathrm{off}},\qquad \eta_{\mathrm{on}}>\eta_{\mathrm{off}}.$$

$$T_{\mathrm{dwell}}\geq\frac{\eta_{\mathrm{on}}-\eta_{\mathrm{off}}}{L_h},\qquad L_h\geq\sup_t|\dot h_b(t)|.$$

Mathematical statement

Let $x$ be the fast attractor state, $h$ the intermediate hysteresis state, and $r$ the slow routing state. Use $\tau_x\dot x=f_a(x)$, $\tau_h\dot h=\sigma(g(r,x))-h$, and $\tau_r\dot r=F(r,u)$, with $\tau_x\ll\tau_h\ll\tau_r$, where $a$ is the active memory index, $u$ is the input, and $\sigma$ is a bounded sigmoid. Switch from memory $a$ to candidate $b$ only when $h_b\geq\eta_{\mathrm{on}}$, and retain $b$ until its score drops below $\eta_{\mathrm{off}}$, with $\eta_{\mathrm{on}}>\eta_{\mathrm{off}}$. The hysteresis width $\Delta\eta=\eta_{\mathrm{on}}-\eta_{\mathrm{off}}$ gives an anti-chattering margin. If the score derivative is bounded by $|\dot h_b|\leq L_h$, a transition requiring traversal of the hysteresis band has dwell time at least $\Delta\eta/L_h$.

Implementation notes

Integration point: use this module as the transition controller in a recurrent language model, latent world model, or regime-switching state-space model. The fast state x is the active attractor representation, r is a GRU-like slow context state, and h is a vector of leaky transition scores. Pseudocode: update r once every M fast steps using r += dt/tau_r*F(r,u); compute candidate scores s=softmax(Linear([r,x])); update h += dt/tau_h*(s-h); retain the current index a unless h[b] exceeds eta_on; after switching, forbid another switch until the active score falls below eta_off. Backpropagate through the leaky updates and use a straight-through estimator only for the discrete index if needed. Add a penalty for switches before the target dwell time. Timescales, thresholds, and the theoretical dwell bound are explicit; estimate L_h from the maximum or high quantile of observed finite-difference derivatives of h, multiplied by a safety factor. First experiment: train a four-regime synthetic sequence predictor with noisy regime-specific dynamics. Compare a GRU, soft attention routing, and the hysteretic router at matched parameter count. Sweep Delta eta and tau_h/tau_x. The prediction is that switch count decreases sharply when Delta eta exceeds score-noise amplitude, and measured minimum dwell time should be at least Delta eta/L_h up to discretization error.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: No measurable effect

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a NumPy hysteretic router MVP with reproducible sweeps. Measured dwell times matched Delta_eta/L within 0.13%, chattering disappeared when the hysteresis width reached the score excursion threshold, and the slow-state RMS gain matched the analytic low-pass prediction within 0.14%. In the controlled regime toy, hysteretic routing reduced MSE from 0.004744 to 0.002220 and reduced route changes from 321 to 207, so the proposed mechanism manifested clearly.

Agent confidence: 9/10

Baseline: Soft score interpolation: MSE 0.0047436553; 321 hard-score switch crossings
Idea: Hysteretic router: MSE 0.0022200007; 207 switches; 53.2% lower MSE; dwell-bound ratios 0.9988-1.0009; low-pass gain relative error <=0.14%

Limitations: The comparison is a deterministic NumPy toy rather than a trained GRU or matched-parameter four-regime predictor. It does not test backpropagation, learned slow routing, GPU behavior, long-horizon language modeling, or robustness across random seeds.

How to run: python3 verify_router.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.18439) Sep 2, 2026

No significant difference vs. the tuned baseline (+0.5%, p=0.7749, wins 3 / 8 paired seeds).

Benchmark: Sequence modeling
Model: routed_transformer
Paired seeds: 8
Baseline mean: 0.906
Idea mean: 0.9106
Effect (Δ): 0.0045 (+0.5%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.7749 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

The benchmark used 400 training and 200 test examples and 10 epochs rather than the larger default dataset/training budget. The router implementation exposes switch counts and maximum finite-difference hysteresis-state derivatives but does not record individual dwell intervals, so the theoretical dwell bound could not be directly tested at neural-network scale. The idea sweep was evaluated, but all three settings were extremely close and the reported idea result used the central setting rather than an independently selected best configuration.

How to run: python3 bench_hysteretic.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 9/10

Built a local Hysteretic Multiscale Sequence Router on the structurally matching sequence track, using identical transformer encoders and expert heads for soft-routing baseline and hysteretic routing. The baseline achieved MSE 0.90604 versus 0.91059 for the idea, with paired delta +0.00455 and permutation p=0.7749; therefore there was no significant win. The observed switch counts varied substantially, but the dwell-bound signature was not confirmed because the implementation recorded derivative magnitudes without per-example dwell intervals.

Baseline: MSE 0.9060439914; best config lr=0.0015, temperature=0.7; 8 seeds
Idea: MSE 0.9105898216; delta=0.12, tau=2.0; 8 seeds; 3 idea settings evaluated; 3/8 paired wins
Mechanism signature: map[confirmed:false delta_eta:0.12 observed_max_L_h:0.2711498141 predicted:dwell >= delta_eta / L_h predicted_min_dwell:0.4425597723 switch_counts:[0 191 0 322 0 174 0 0]]
Paired delta: idea - baseline = +0.0045458302 MSE
Permutation p value: 0.7749

Benchmark result

No significant difference vs. the tuned baseline (+0.5%, p=0.7749, wins 3 / 8 paired seeds).

Benchmark: Sequence modeling
Model: routed_transformer
Paired seeds: 8
Baseline mean: 0.906
Idea mean: 0.9106
Effect (Δ): 0.0045 (+0.5%; negative = idea better)
Wins: 3 / 8 paired seeds
p-value: 0.7749 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: No effect
Baseline tuning: swept over 3 configs

Limitations:

The benchmark used 400 training and 200 test examples and 10 epochs rather than the larger default dataset/training budget. The router implementation exposes switch counts and maximum finite-difference hysteresis-state derivatives but does not record individual dwell intervals, so the theoretical dwell bound could not be directly tested at neural-network scale. The idea sweep was evaluated, but all three settings were extremely close and the reported idea result used the central setting rather than an independently selected best configuration.

How to run: python3 bench_hysteretic.py

Artifacts

Implementation overview ⬇ Download all as ZIP 6 files · code, reports and structured results

🐍 bench_hysteretic.py 3.7 KB View Raw
📊 bench_report.json 3.3 KB View Raw
📄 report.md 1.2 KB View
📄 report_bench_2026-08-31T121847.md 4.6 KB View
📊 results.json 2.5 KB View Raw
🐍 verify_router.py 5.4 KB View Raw

Idea description

Formulas

Mathematical statement

Implementation notes

Verification

Artifacts

Related ideas