Schäffer-Covariant Isometric Recurrent Layer

Integrate this into an RNN or linear state-space block. Let the trainable base transition be $X\in\mathbb R^{d\times d}$ and enforce contraction with spectral normalization, $X=\widetilde X/\max(1,\sigma_{\max}(\widetilde X))$, or use a differentiable contraction parameterization. For a truncation length $K$, maintain hidden state $(h,q_0,\ldots,q_{K-1})$, where $h\in\mathbb R^d$ and each $q_j$ is a defect-memory vector. At every timestep compute $D_X=(I-X^\top X+\delta I)^{1/2}$ by an eigendecomposition for small $d$, or approximate its action with a Newton-Schulz iteration; update $h'=Xh+Bu_t$, $q'_0=D_Xh$, and $q'_j=q_{j-1}$ for $1\le j<K$. The exact infinite lift is isometric; truncation discards $q_{K-1}$, so log the discarded energy and compare it with an ordinary RNN. To test covariance, create a second learned channel $V_2$ and set $V_1$ to the block transition or its lifted matrix. Use $f(V_1)=V_1^m$ for $m=2$ initially, adding $\lambda\mathcal L_{\mathrm{cov}}$ to the task loss. The paper supplies the defect-lift construction and covariance identity; empirical choices are $K$, $m$, contraction normalization, and $\lambda$. First run a small permuted-MNIST or sequential-CIFAR experiment with $d=64$ and $K\in\{4,8,16\}$ against a spectrally normalized vanilla RNN and an orthogonal RNN at equal parameter count. Measure gradient norm growth, hidden-state norm drift, long-horizon accuracy, and loss after 10k updates. Success means substantially fewer exploding or vanishing gradients and better accuracy when sequence length increases, without more than 20% extra FLOPs.