Retained-Excess Recurrent Unit

1. Integration point: implement this as a drop-in recurrent cell or diagonal state-space layer. Keep the ordinary hidden state h_t, but add an excess state e_t with the same dimensionality as the clipped preactivation. For vector outputs, use elementwise clipping and either a learned scalar rho, a learned vector sigmoid(rho), or a diagonal matrix. Apply clipping only to the externally exposed state or prediction; retain e_t internally. 2. Pseudocode: compute q = W_h @ h_prev + W_x @ u + b + rho * e_prev; y = clamp(q, -C, C); e = q - y; h = tanh(y) or tanh(q), depending on whether bounded dynamics are desired; return h, y, e. Initialize e to zero, constrain rho with sigmoid, and optionally regularize rho toward values below one. 3. Computed quantities are q, y, and e exactly; estimate the effective innovation tail index alpha using a Hill estimator on q-rho*e_prev, and estimate the empirical spectral radius of W_h. The paper supplies the clipping/hidden-excess mechanism and the tail-response prediction; the neural recurrence and nonlinear feature map are the adaptation. 4. First cheap experiment: train vanilla tanh RNN, GRU, and this cell on sequential MNIST and synthetic heavy-tailed autoregressive forecasting with Student-t innovations. Sweep C and rho in {0,0.25,0.5,0.75,0.9}. Measure conditional mean q_{t+1} after q_t>C, same-side saturation probability, and opposite-side probability. The falsifiable signature is that the post-saturation mean grows approximately linearly with C and rho, is near zero at rho=0, while opposite-side transitions decay approximately as C^{-alpha}; fit log probability versus log C and require slope near -alpha within 20%. Also test whether long-horizon forecasting improves specifically on sequences containing saturation events, not merely on Gaussian data.