# Retained-Excess Recurrent Unit

- ID: 2004
- Canonical URL: https://synthcore.org/idea/2004/retained-excess-recurrent-unit
- API JSON: https://synthcore.org/api/idea/2004.json
- API Markdown: https://synthcore.org/api/idea/2004.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.08625](https://arxiv.org/abs/2608.08625)
- Category: dynamics
- Solves: stability, accuracy, memory
- ML areas: rnn, ssm, training-dynamics
- Math tags: dynamical-systems, stochastic-processes, probability, statistics
- Ratings: usefulness 7/10; difficulty 5/10; novelty 7/10

## Idea description

Replace a memoryless clipped recurrent output with a clipped observable plus a latent retained overshoot. The network exposes only a bounded output, but stores a fraction of the amount that would have exceeded the bound and feeds it into the next hidden-state update, allowing the model to represent persistent post-saturation effects without making the visible output unstable.

## Mathematical statement

Let z_t be the unconstrained scalar or vector preactivation, y_t the bounded output, C>0 the componentwise clipping limit, and e_t=z_t-y_t the hidden excess. Define y_t=clip(z_t,-C,C) and e_t=z_t-y_t. For an input-driven recurrent unit, use z_t=W_h h_{t-1}+W_x u_t+b+rho e_{t-1}, h_t=phi(z_t), where u_t is the input, h_t the hidden state, phi is a bounded or ordinary feature map, and 0<=rho<1 is the retained-excess coefficient. If the exogenous innovations epsilon_t in z_t=epsilon_t+rho e_{t-1} are independent and regularly varying with tail index alpha>1, then the stationary excess inherits tail index alpha while its amplitude is increased by retention. After y_t=C, the conditional next-step mean contribution from memory is positive, approximately rho E[e_t|z_t>C], and for Pareto-like tails scales as C; the opposite-limit event requires a compensating negative innovation and is asymptotically suppressed as a power of C.

## Key formulas

- $$y_t=\operatorname{clip}(z_t,-C,C)=\min\{C,\max\{-C,z_t\}\},\qquad e_t=z_t-y_t,$$
- $$z_t=W_hh_{t-1}+W_xu_t+b+\rho e_{t-1},\qquad h_t=\phi(z_t),\qquad 0\leq\rho<1,$$
- $$\mathbb{E}[z_{t+1}\mid z_t>C]-\mathbb{E}[z_{t+1}\mid z_t<-C]\approx 2\rho\,\mathbb{E}[e_t\mid z_t>C]>0,$$
- $$\Pr(y_{t+1}=-C\mid y_t=C)\sim K C^{-\alpha},\qquad \mathbb{E}[e_t\mid z_t>C]\propto C\quad(\alpha>1),$$

## Implementation notes

1. Integration point: implement this as a drop-in recurrent cell or diagonal state-space layer. Keep the ordinary hidden state h_t, but add an excess state e_t with the same dimensionality as the clipped preactivation. For vector outputs, use elementwise clipping and either a learned scalar rho, a learned vector sigmoid(rho), or a diagonal matrix. Apply clipping only to the externally exposed state or prediction; retain e_t internally. 2. Pseudocode: compute q = W_h @ h_prev + W_x @ u + b + rho * e_prev; y = clamp(q, -C, C); e = q - y; h = tanh(y) or tanh(q), depending on whether bounded dynamics are desired; return h, y, e. Initialize e to zero, constrain rho with sigmoid, and optionally regularize rho toward values below one. 3. Computed quantities are q, y, and e exactly; estimate the effective innovation tail index alpha using a Hill estimator on q-rho*e_prev, and estimate the empirical spectral radius of W_h. The paper supplies the clipping/hidden-excess mechanism and the tail-response prediction; the neural recurrence and nonlinear feature map are the adaptation. 4. First cheap experiment: train vanilla tanh RNN, GRU, and this cell on sequential MNIST and synthetic heavy-tailed autoregressive forecasting with Student-t innovations. Sweep C and rho in {0,0.25,0.5,0.75,0.9}. Measure conditional mean q_{t+1} after q_t>C, same-side saturation probability, and opposite-side probability. The falsifiable signature is that the post-saturation mean grows approximately linearly with C and rho, is near zero at rho=0, while opposite-side transitions decay approximately as C^{-alpha}; fit log probability versus log C and require slope near -alpha within 20%. Also test whether long-horizon forecasting improves specifically on sequences containing saturation events, not merely on Gaussian data.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a retained-excess recurrent update with elementwise clipping, latent excess retention, heavy-tailed toy verification, and a stable forecasting comparison in experiment.py. The mechanism manifested partially: rho=0 produced near-zero post-saturation means, and excess/post-saturation response scaled nearly linearly with C (log-log slopes 0.96–1.07); however, opposite-limit transition slopes were about -2.0 to -2.17 rather than the predicted -2.5 within the stated 20% tolerance. Forecasting showed no meaningful practical win: MSE changed from 3.335733 to 3.335610 and saturated-event MSE from 17.089357 to 17.088820.

### Mechanism check

- Verdict: Built a retained-excess recurrent update with elementwise clipping, latent excess retention, heavy-tailed toy verification, and a stable forecasting comparison in experiment.py. The mechanism manifested partially: rho=0 produced near-zero post-saturation means, and excess/post-saturation response scaled nearly linearly with C (log-log slopes 0.96–1.07); however, opposite-limit transition slopes were about -2.0 to -2.17 rather than the predicted -2.5 within the stated 20% tolerance. Forecasting showed no meaningful practical win: MSE changed from 3.335733 to 3.335610 and saturated-event MSE from 17.089357 to 17.088820.
- Confidence: 8/10
- Limitations: The neural RNN training comparison was abandoned after repeated NaNs and replaced with a stable least-squares one-step forecasting comparison. No sequential MNIST, GRU comparison, learned rho, multivariate recurrence, confidence intervals, or broad hyperparameter search was tested.

## Artifacts

- [experiment.py](https://synthcore.org/code/763/experiment.py)
- [forecast_results.json](https://synthcore.org/code/763/forecast_results.json)
- [report.md](https://synthcore.org/code/763/report.md)
- [results.json](https://synthcore.org/code/763/results.json)
- [toy_results.json](https://synthcore.org/code/763/toy_results.json)
- [Download all files as ZIP](https://synthcore.org/download/763)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.