# Bounded Signed Fast-Memory Gate

- ID: 364
- Canonical URL: https://synthcore.org/idea/364/bounded-signed-fast-memory-gate
- API JSON: https://synthcore.org/api/idea/364.json
- API Markdown: https://synthcore.org/api/idea/364.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.02363](https://arxiv.org/abs/2607.02363)
- Category: dynamics
- Solves: stability, accuracy, scalability
- ML areas: training-dynamics, rnn, mlp
- Math tags: dynamical-systems, linear-algebra, stability
- Ratings: usefulness 7/10; difficulty 4/10; novelty 5/10

## Idea description

Replace an unconstrained input-dependent multiplier on a recurrent fast-weight state with a sign-preserving tanh gate. The new state retains an additive low-rank update and optionally a separately modulated innovation term, but the accumulated-memory branch can never be amplified by a factor whose magnitude exceeds one.

## Mathematical statement

The QFWP controller produces a hidden vector h_t=\phi_{\Omega}(x_t)\in\mathbb{R}^H, two affine heads \ell_t=W_\ell h_t+b_\ell\in\mathbb{R}^L and r_t=W_rh_t+b_r\in\mathbb{R}^Q, and the rank-one update \Delta_t=\ell_t r_t^    op\in\mathbb{R}^{L    imes Q}. The baseline recurrence is \Theta_t=\Theta_{t-1}+\Delta_t, where \Theta_t\in\mathbb{R}^{L    imes Q}. For a neural-network adaptation, generate an old-state multiplier a_t from the controller and define the sign-preserving bounded function \operatorname{sptanh}(a)=\operatorname{sign}(a)    anh(|a|), so |\operatorname{sptanh}(a)|<1. Use \Theta_t=G_t\odot\Theta_{t-1}+U_t\odot\Delta_t, where G_t is a broadcast or elementwise \operatorname{sptanh}(a_t), U_t is an optional unconstrained or sigmoid-bounded new-update gate, and \odot is elementwise multiplication. The key dynamical property is |G_{t,i}|<1, which removes the possibility that repeated old-state multipliers amplify memory geometrically; without this constraint, a recurrence containing \alpha_t\Theta_{t-1} can grow like \prod_{j=1}^t|\alpha_j|.

## Key formulas

- $$h_t=\phi_{\Omega}(x_t)\in\mathbb{R}^{H},\qquad \ell_t=W_{\ell}h_t+b_{\ell}\in\mathbb{R}^{L},\qquad r_t=W_rh_t+b_r\in\mathbb{R}^{Q}$$
- $$\Delta_t=\ell_t r_t^{\top}\in\mathbb{R}^{L\times Q},\qquad \Theta_t=\Theta_{t-1}+\Delta_t$$
- $$\operatorname{sptanh}(a)=\operatorname{sign}(a)\tanh(|a|),\qquad |\operatorname{sptanh}(a)|<1$$
- $$\Theta_t=G_t\odot\Theta_{t-1}+U_t\odot\Delta_t,\qquad G_t=\operatorname{sptanh}(W_g h_t+b_g),\quad U_t=\sigma(W_u h_t+b_u)$$

## Implementation notes

Integrate this at the fast-weight or hypernetwork state update, not in the base model weights. Start with a small recurrent fast-weight module: an MLP controller receives token embedding x_t or the current sequence feature, produces h_t, and emits low-rank factors l_t and r_t for an update to a matrix Theta_t used by a linear readout, recurrent transition, or attention value projection. Add a separate old-memory head a_t=W_g h_t+b_g and compute G_t=sign(a_t)*tanh(abs(a_t)); use epsilon-stable code such as sign(a_t)*tanh(abs(a_t)) with sign(0)=0. Broadcast a vector G_t over rows or columns of Theta_t, or use a full low-rank gate G_t=u_t v_t^T passed through sptanh for a cheaper alternative. Compute Delta_t=l_t[:,None]*r_t[None,:], then update Theta=G*Theta+sigmoid(W_u h+b_u)*Delta. Detach nothing: both gates and fast weights should receive gradients through time initially. Compare four variants: additive baseline Theta=Theta+Delta, unconstrained old gate, bounded ordinary tanh gate, and sign-preserving tanh gate. Log ||Theta_t||_F, the maximum per-step amplification ratio ||G_t odot Theta_{t-1}||/||Theta_{t-1}||, gradient norms, and loss by sequence position. The math supplies the bounded-gate construction; estimate all gate parameters by backpropagation and initialize b_g near arctanh(0.9) if positive retention is desired. First experiment: train a 2-layer GRU-like fast-weight predictor or a tiny Transformer with a 64x64 fast matrix on sequential MNIST, permuted MNIST, or a synthetic copy task with lengths 128, 512, and 2048. The success signal is no divergence at long lengths, lower gradient variance, and lower validation loss than the unconstrained gated model at equal parameter count and FLOPs.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built and ran a recurrent scalar fast-memory MVP with additive, unconstrained, tanh, and sign-preserving-tanh updates. The math check verified |sptanh(a)|<1 and showed a 40-step multiplier product of 0.00069 versus 1469.77 for an unconstrained multiplier of 1.2. On the synthetic AR task, sptanh reached loss 0.1125 at length 256 versus 0.1023 for ordinary tanh and 0.1790 for unconstrained gating; therefore the stability property was real, but the promised accuracy win over the relevant gated baselines was not observed.

### Mechanism check

- Verdict: Built and ran a recurrent scalar fast-memory MVP with additive, unconstrained, tanh, and sign-preserving-tanh updates. The math check verified |sptanh(a)|<1 and showed a 40-step multiplier product of 0.00069 versus 1469.77 for an unconstrained multiplier of 1.2. On the synthetic AR task, sptanh reached loss 0.1125 at length 256 versus 0.1023 for ordinary tanh and 0.1790 for unconstrained gating; therefore the stability property was real, but the promised accuracy win over the relevant gated baselines was not observed.
- Confidence: 9/10
- Limitations: The MVP uses a scalar fast-memory state rather than a full matrix fast-weight module, one synthetic AR task, 120 training steps, and a single seed per variant. It did not measure gradient variance, FLOPs, wall-clock speed, or lengths 512/2048. Also, sign(a)*tanh(abs(a)) is algebraically identical to tanh(a), so the proposed sign-preserving form is not a distinct function from ordinary tanh.

## Artifacts

- [report.md](https://synthcore.org/code/72/report.md)
- [results.json](https://synthcore.org/code/72/results.json)
- [run_experiment.py](https://synthcore.org/code/72/run_experiment.py)
- [Download all files as ZIP](https://synthcore.org/download/72)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.