# Normal Damped-Rotation Recurrent Layer

- ID: 1765
- Canonical URL: https://synthcore.org/idea/1765/normal-damped-rotation-recurrent-layer
- API JSON: https://synthcore.org/api/idea/1765.json
- API Markdown: https://synthcore.org/api/idea/1765.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.04028](https://arxiv.org/abs/2608.04028)
- Category: architecture
- Solves: stability, accuracy, sample-efficiency
- ML areas: rnn, ssm, training-dynamics
- Math tags: dynamical-systems, linear-algebra, spectral-theory
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace an unconstrained recurrent matrix with an orthogonally mixed block diagonal matrix whose blocks are independently parameterized damped rotations. The model receives explicit phase mixing from the rotation frequencies and controlled forgetting from the decay rates, while its linear recurrent dynamics have a known contraction factor before the nonlinear activation.

## Mathematical statement

The paper constructs each two-dimensional mode from the continuous generator A_mj = -gamma_mj I + omega_mj J, where I is the 2-by-2 identity, J = [[0,-1],[1,0]] is the infinitesimal rotation generator, gamma_mj >= 0 is the damping rate, and omega_mj is the angular frequency. Exact unit-time discretization gives B_mj = exp(A_mj) = exp(-gamma_mj) R(omega_mj), where R(theta) = [[cos(theta),-sin(theta)],[sin(theta),cos(theta)]]. For an orthogonal mixing matrix Q_m, the full recurrent operator is W_m = Q_m blkdiag(B_mj) Q_m^T. Its eigenvalues satisfy |lambda_mj| = exp(-gamma_mj), arg(lambda_mj) = plus or minus omega_mj, and its spectral norm is ||W_m||_2 = exp(-min_j gamma_mj) < 1. This means the recurrent linear map is normal and has an explicit contraction margin determined only by the slowest-decaying mode. With tanh, the global Jacobian norm is no larger than this linear norm because tanh has derivative magnitude at most one.

## Key formulas

- $$A_{mj}=-\gamma_{mj}I+\omega_{mj}J,\qquad B_{mj}=e^{A_{mj}}=e^{-\gamma_{mj}}R(\omega_{mj})$$
- $$W_m=Q_m\operatorname{blkdiag}(B_{mj})Q_m^{\top},\qquad |\lambda_{mj}|=e^{-\gamma_{mj}},\qquad \arg(\lambda_{mj})=\pm\omega_{mj}$$
- $$\|W_m\|_2=e^{-\min_j\gamma_{mj}}<1$$
- $$z_{t+1}=\tanh\!\left(W_m z_t+W_{\mathrm{in}}u_t+b\right),\qquad z_0=0$$

## Implementation notes

Integrate this at the recurrent state transition of a tanh RNN or GRU-like minimal baseline. Choose hidden width d = 2K. Store K unconstrained frequency parameters omega_j and positive decay parameters gamma_j = softplus(rho_j) + gamma_min. Construct each 2-by-2 block B_j = exp(-gamma_j) [[cos(omega_j), -sin(omega_j)], [sin(omega_j), cos(omega_j)]]. Either use Q = I for the cheapest version or generate one fixed random Gaussian matrix and orthogonalize it once with QR; set W = Q blockdiag(B_j) Q^T. At every timestep compute preactivation a_t = W h_t + W_in x_t + b and h_{t+1} = tanh(a_t). For a trainable recurrent layer, optimize W_in, b, and optionally Q through a Cayley or QR parameterization, and optimize rho_j and omega_j; for a reservoir, freeze W, W_in, and b and train only a linear readout on [h_t, x_t, 1]. Pseudocode: initialize gamma_j and omega_j; build B_j; build W; for each batch and timestep, h = tanh(W @ h + W_in @ x + b); accumulate readout loss. The formula ||W||_2 = exp(-min gamma_j) is computed analytically and should be logged as the predicted linear stability margin; estimate the actual nonlinear Jacobian norm with occasional power iteration. First test on sequential MNIST and NARMA-10 using width 128, comparing vanilla tanh RNN, orthogonal RNN, and this layer at equal parameters and FLOPs. Success means fewer exploding or saturated trajectories, a lower validation loss at equal training steps, and stable performance when sequence length increases from 100 to 1000.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: harms
- Verdict: Built a fixed-reservoir Normal Damped-Rotation layer using orthogonal similarity transforms of independently damped 2D rotations. The mechanism checks succeeded: spectral-norm and eigenvalue predictions matched to approximately 1e-15, decay-to-e-fold and epsilon-crossing predictions matched exactly up to integer rounding, and all sampled tanh Jacobians stayed below the predicted contraction bound. In the small NARMA-10 comparison, the structured reservoir achieved lower test NRMSE than the scaled random recurrent baseline, although this was only one tiny setup.

### Mechanism check

- Verdict: Built a fixed-reservoir Normal Damped-Rotation layer using orthogonal similarity transforms of independently damped 2D rotations. The mechanism checks succeeded: spectral-norm and eigenvalue predictions matched to approximately 1e-15, decay-to-e-fold and epsilon-crossing predictions matched exactly up to integer rounding, and all sampled tanh Jacobians stayed below the predicted contraction bound. In the small NARMA-10 comparison, the structured reservoir achieved lower test NRMSE than the scaled random recurrent baseline, although this was only one tiny setup.
- Confidence: 9/10
- Limitations: Only a small fixed-reservoir NARMA-10 experiment was tested; no sequential MNIST, NARMA-20, long-sequence sweep, trained RNN/GRU comparison, FLOP measurement, multiple-seed confidence intervals, or optional trainable orthogonal mixing was evaluated.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [bench_experiment.py](https://synthcore.org/code/661/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/661/bench_report.json)
- [experiment.py](https://synthcore.org/code/661/experiment.py)
- [report.md](https://synthcore.org/code/661/report.md)
- [report_bench_2026-08-31T212054.md](https://synthcore.org/code/661/report_bench_2026-08-31T212054.md)
- [results.json](https://synthcore.org/code/661/results.json)
- [Download all files as ZIP](https://synthcore.org/download/661)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.