# Threshold-Projection Recurrent Memory

- ID: 908
- Canonical URL: https://synthcore.org/idea/908/threshold-projection-recurrent-memory
- API JSON: https://synthcore.org/api/idea/908.json
- API Markdown: https://synthcore.org/api/idea/908.md
- Verification status: unverified
- Source: [arXiv:2607.14321](https://arxiv.org/abs/2607.14321)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: rnn, ssm, world-model
- Math tags: geometry, dynamical-systems, optimization
- Ratings: usefulness 6/10; difficulty 4/10; novelty 7/10

## Idea description

Replace or augment a recurrent cell with multiple hysteresis memory branches whose states remain unchanged while the input stays within a branch-specific radius, then move toward the current input only when that radius is exceeded. The resulting cell has explicit persistence and bounded state changes, giving it an inductive bias for temporal hysteresis and reducing the need for the network to learn long-term memory behavior from scratch.

## Mathematical statement

The paper's explicit hysteresis update is a projection rule for each branch n. Given current input vector \(\mathbf h\in\mathbb R^d\), previous branch state \(\mathbf h_{\mathrm{rev},p}^{n}\in\mathbb R^d\), and threshold radius \(\kappa^n>0\), the new state is \(\mathbf h_{\mathrm{rev}}^{n}=\mathbf h_{\mathrm{rev},p}^{n}\) when \(\|\mathbf h-\mathbf h_{\mathrm{rev},p}^{n}\|_2\leq\kappa^n\), otherwise it is \(\mathbf h-\kappa^n(\mathbf h-\mathbf h_{\mathrm{rev},p}^{n})/\|\mathbf h-\mathbf h_{\mathrm{rev},p}^{n}\|_2\). Equivalently, \(\mathbf h_{\mathrm{rev}}^{n}\) is the Euclidean projection of the previous state onto the closed ball centered at \(\mathbf h\) with radius \(\kappa^n\): \(\Pi_{B(\mathbf h,\kappa^n)}(\mathbf h_{\mathrm{rev},p}^{n})\). The paper aggregates branches as \(\mathbf h_{\mathrm{rev}}=\sum_{n=1}^{N}w^n\mathbf h_{\mathrm{rev}}^n\), where \(w^n\geq0\) and \(\sum_n w^n=1\). The projection bounds every branch's distance from the current input by \(\kappa^n\), while convex aggregation keeps the combined state inside the corresponding weighted geometric envelope. For neural networks, use \(\kappa^n=\operatorname{softplus}(\rho^n)+\epsilon\), \(w^n=\operatorname{softmax}(a)_n\), and optionally learn an input projection before applying the rule.

## Key formulas

- $$\mathbf{h}_{\mathrm{rev}}^{n}=\begin{cases}\mathbf{h}_{\mathrm{rev},\mathrm{p}}^{n}&\text{if }\left\lVert\mathbf{h}-\mathbf{h}_{\mathrm{rev},\mathrm{p}}^{n}\right\rVert_2\leq\kappa^{n},\\[2mm]\mathbf{h}-\kappa^{n}\frac{\mathbf{h}-\mathbf{h}_{\mathrm{rev},\mathrm{p}}^{n}}{\left\lVert\mathbf{h}-\mathbf{h}_{\mathrm{rev},\mathrm{p}}^{n}\right\rVert_2}&\text{otherwise,}\end{cases}$$
- $$\mathbf{h}_{\mathrm{rev}}=\sum_{n=1}^{N}w^{n}\mathbf{h}_{\mathrm{rev}}^{n},\qquad w^{n}\geq0,\qquad\sum_{n=1}^{N}w^{n}=1,$$
- $$\mathbf{h}_{\mathrm{rev}}^{n}=\Pi_{B(\mathbf h,\kappa^n)}\!\left(\mathbf{h}_{\mathrm{rev},p}^{n}\right),\qquad B(\mathbf h,\kappa^n)=\{\mathbf z:\|\mathbf z-\mathbf h\|_2\leq\kappa^n\},$$
- $$\mathbf y_t=\operatorname{MLP}\!\left([\mathbf x_t,\,\mathbf m_t,\,\mathbf m_t-\mathbf m_{t-1}]\right),\qquad \mathbf m_t=\sum_{n=1}^{N}w_n\,\Pi_{B(\mathbf u_t,\kappa_n)}(\mathbf m_{t-1}^{n}),\quad \mathbf u_t=W_x\mathbf x_t+b_x,$$

## Implementation notes

(1) Integration point: implement a PyTorch module `HysteresisMemory` for sequential data. At timestep t, map the input tensor `x_t` with shape `[batch,d_in]` to `u_t = W_x @ x_t + b_x` with shape `[batch,d]`; maintain N state tensors `m_prev[n]` of shape `[batch,d]`. This can replace a GRU hidden-state update, sit before an SSM/readout layer, or run in parallel to a standard recurrent layer. (2) Pseudocode: initialize every `m_prev[n]` to zero; compute `kappa = softplus(rho)+1e-4` and `w = softmax(a)`; for each timestep and branch compute `delta = u_t - m_prev[n]`, `dist = sqrt(sum(delta**2, dim=-1, keepdim=True)+1e-8)`, and `candidate = u_t - kappa[n]*delta/dist`; set `m_new[n] = where(dist <= kappa[n], m_prev[n], candidate)`; aggregate `m_t = sum_n w[n]*m_new[n]`; predict with `head(concat(x_t,m_t,m_t-m_prev_aggregate))`; then assign `m_prev = m_new`. The displayed paper formula is exactly the conditional update and radial projection. (3) Mathematics versus estimation: the projection rule, Euclidean-ball geometry, and normalized branch weights are fixed by the construction. Learn `rho`, `a`, `W_x`, and the prediction head by backpropagation. Estimate a useful initialization for each radius from displacement quantiles of `u_t-u_{t-1}`, for example initialize `kappa[n]` to quantiles from 0.1 to 0.9. A smooth training variant uses `g=sigmoid((dist-kappa)/tau)` and `m_new=(1-g)*m_prev+g*candidate`; anneal `tau` and finally evaluate the hard rule. (4) First cheap experiment: train on Moving-MNIST frame prediction or an electricity/load forecasting dataset using a one-layer GRU with matched hidden size and approximate FLOPs as baseline. Compare GRU, a vanilla leaky integrator, and this module for N in {4,8,16}. Measure validation MSE, long-horizon error, performance under repeated or reversed input segments, gradient norms over 1,000-step sequences, and inference throughput. Success means lower long-horizon error or a smaller hidden state at equal error, with fewer exploding or vanishing-gradient failures and stable predictions when the input reverses direction.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.