Reversible Two-Relaxation Recurrent Block

Integrate this as the recurrent transition in a small RNN, linear state-space model, or sequence mixer. Store the hidden state with shape [batch, positions, directions, features], and arrange directions in opposite pairs. Implement R as a fixed channel permutation. For every state f, compute f_plus = (f + R(f))/2 and f_minus = (f - R(f))/2, then apply complementary relaxation: g = f - s_plus*f_plus - s_minus*f_minus. Parameterize s_minus = 2*sigmoid(a) and set s_plus = 2 - s_minus, either globally or per feature. Apply U after relaxation. An exact low-cost U can be a fixed cyclic shift over positions followed by learned orthogonal feature mixing; parameterize the mixer with Householder reflections or obtain its orthogonal factor using QR retraction after each optimizer step. The recurrent update is f_next = U(g) + B x_t, followed by a nonlinear readout outside the recurrent transition. For diagnostics, estimate ||A^k|| by automatic-differentiation Jacobian-vector products and 10–20 power iterations. Approximate the numerical range by sampling normalized random vectors v_j and recording v_j^* A v_j; fit an empirical ellipse and penalize samples outside the unit disk or with ellipse radius above one. The exact mathematical ingredients are the parity split, complementary rates, orthogonality, and reversibility; ellipse parameters are estimated empirically. First test on a 1,000-step copy task and Tiny Shakespeare using a 2-layer, 128-dimensional model. Compare vanilla tanh RNN, orthogonal RNN, and this block at matched parameter count and FLOPs. Record gradient norm versus sequence length, maximum hidden-state amplification, validation loss, and time-to-target loss. Success means bounded amplification and non-diverging gradients at lengths where the vanilla RNN fails, with equal or lower perplexity and less than 10% extra compute.