# Entire Bilinear-Orthogonal Neural Flow

- ID: 1664
- Canonical URL: https://synthcore.org/idea/1664/entire-bilinear-orthogonal-neural-flow
- API JSON: https://synthcore.org/api/idea/1664.json
- API Markdown: https://synthcore.org/api/idea/1664.md
- Verification status: unverified
- Source: [arXiv:2608.01253](https://arxiv.org/abs/2608.01253)
- Category: dynamics
- Solves: stability, memory, generalization
- ML areas: rnn, ssm, training-dynamics
- Math tags: linear-algebra, dynamical-systems, geometry, complex-analysis
- Ratings: usefulness 5/10; difficulty 5/10; novelty 5/10

## Idea description

Replace an unconstrained recurrent or state-space transition with a complex-orthogonal flow generated by a skew-transpose matrix. The transition preserves a bilinear quadratic quantity exactly, preventing repeated application across long sequences from causing norm explosion or decay in the linear dynamics.

## Mathematical statement

The paper defines the entire one-parameter group W_z=e^{zK}, where z is complex time and K is a complex d by d generator. Complex orthogonality means W_z^T W_z=I_d, where T is transpose without complex conjugation. Differentiating this identity at z=0 gives K^T=-K. Therefore, for u(z)=W_z u(0), the bilinear energy u(z)^T u(z) equals u(0)^T u(0). A trainable low-rank skew generator can be parameterized as K=A B^T-B A^T, with A and B in C^{d by r}. For cheaper discretization, use the Cayley transform C_tau(K)=(I_d-tau K/2)^{-1}(I_d+tau K/2). If K^T=-K and the inverse exists, then C_tau(K)^T C_tau(K)=I_d, so each discrete transition preserves the same bilinear form. A real implementation represents a complex vector u=a+i b by the real vector [a;b] and represents K=K_r+iK_i by the real block matrix [[K_r,-K_i],[K_i,K_r]].

## Key formulas

- $$W_z=e^{zK},\qquad z\in\mathbb{C}.$$
- $$W_z^T W_z=I_d,\qquad K^T=-K.$$
- $$u(z)^T u(z)=u(0)^T u(0),\qquad u(z)=W_z u(0).$$
- $$K=A B^T-B A^T,\qquad C_\tau(K)=\left(I_d-\frac{\tau}{2}K\right)^{-1}\left(I_d+\frac{\tau}{2}K\right).$$

## Implementation notes

Integrate the construction at the hidden-state transition of an RNN or state-space model. Maintain trainable complex matrices A and B with shapes d by r, construct K=A transpose(B)-B transpose(A), and apply either an exact exponential for small hidden size or a Cayley step for larger models. Pseudocode is: K = A @ transpose(B) - B @ transpose(A); M_left = I - tau*K/2; M_right = I + tau*K/2; h_linear = solve(M_left, M_right @ h); h_next = h_linear + alpha*tanh(V @ h_linear + c). Initialize alpha near zero so training starts from the stable linear flow. In ordinary real-valued PyTorch or JAX, store real and imaginary parts separately. Convert K=K_r+iK_i into the real block matrix [[K_r,-K_i],[K_i,K_r]], concatenate the real and imaginary parts of h, and use a batched linear solve. The skew condition is guaranteed algebraically by the parameterization; numerical preservation should be measured by the relative drift in h^T h after one step and after a long rollout. Also estimate the largest singular value of the implemented transition empirically. First run a cheap comparison on sequential MNIST and permuted sequential MNIST using equal hidden width: vanilla RNN, GRU, unconstrained linear state-space transition, and the proposed Cayley flow. Track validation accuracy, loss at equal FLOPs, hidden-state norm over 1,000 steps, gradient norms, and wall-clock time. Success is stable long-horizon gradients with no systematic hidden-state growth or decay, plus accuracy matching or exceeding the vanilla RNN; a secondary success criterion is lower memory than an ODE solver while retaining comparable sequence accuracy.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.