# Inertia-Constrained Kreiss RNN

- ID: 2853
- Canonical URL: https://synthcore.org/idea/2853/inertia-constrained-kreiss-rnn
- API JSON: https://synthcore.org/api/idea/2853.json
- API Markdown: https://synthcore.org/api/idea/2853.md
- Verification status: unverified
- Source: [arXiv:2608.29823](https://arxiv.org/abs/2608.29823)
- Category: dynamics
- Solves: stability, accuracy
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: spectral-theory, linear-algebra, dynamical-systems
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Replace an unconstrained recurrent transition matrix with a J-selfadjoint matrix A, where J is a fixed diagonal signature matrix with only a small number of negative entries. Add a sampled Kreiss-resolvent penalty to suppress transient amplification while preserving the expressive dimension of the hidden state. The paper's bound predicts that worst finite-time amplification depends on the smaller inertia index rather than the full hidden dimension.

## Mathematical statement

The main theorem states that if A is an n by n matrix with spectrum in the closed unit disk, J is a fundamental symmetry with inertia (n-q,q), and A*J=JA, then M(A) is at most e times r(A) times K(A), where M(A)=sup over k greater than or equal to zero of ||A^k||_2, K(A)=sup over |z| greater than 1 of (|z|-1)||(zI-A)^(-1)||_2, and r(A)=min{d(A),2q+1,2(n-q)+1}. Here d(A) is the degree of A's minimal polynomial, n is the hidden dimension, and q is the smaller inertia index. The constraint is implemented by A=JH with H=H*, because (JH)*J=HJ J=H=JA. The theorem's transferable property is that small q yields r(A) at most 2q+1 independently of n. In a neural implementation, K(A) is approximated on a finite grid of complex points outside the unit disk; this is a practical surrogate rather than a rigorous continuous supremum.

## Key formulas

- $$A^*J=JA,\qquad J=J^*=J^{-1},\qquad \operatorname{inertia}(J)=(n-q,q).$$
- $$\mathcal{M}(A):=\sup_{k\geq 0}\lVert A^k\rVert_2,\qquad \mathcal{K}(A):=\sup_{|z|>1}(|z|-1)\left\lVert(zI-A)^{-1}\right\rVert_2.$$
- $$\mathcal{M}(A)\leq e\min\{d(A),2q+1,2(n-q)+1\}\mathcal{K}(A),\qquad \sigma(A)\subset\overline{\mathbb{D}}.$$
- $$A=JH,\quad H=H^*,\qquad \mathcal{L}_{\mathrm{K}}(A)=\frac{1}{|\mathcal{Z}|}\sum_{z\in\mathcal{Z}}\left[(|z|-1)\,\sigma_{\max}\left((zI-A)^{-1}\right)\right]^2.$$

## Implementation notes

(1) Integration point: use this matrix as the transition operator of a vanilla RNN or linear state-space layer, h_(t+1)=A h_t+B u_t. Choose hidden width n and a small inertia index q, for example q=4, and fix J=diag(+1,...,+1,-1,...,-1). Store an unconstrained real matrix H0, symmetrize it as H=(H0+H0^T)/2, and set A=JH. For complex-valued models use conjugate transpose instead. (2) Pseudocode: construct A=J(H0+H0^T)/2; estimate the spectral radius rho_hat(A) with several Arnoldi or power iterations; set A_s=A*min(1,(1-epsilon)/max(rho_hat(A),epsilon)); sample z=rho_i exp(i theta_j), with rho_i in {1.01,1.05,1.2,1.5,2.0}; estimate sigma_max((zI-A_s)^(-1)) using a few power iterations, each requiring linear solves with zI-A_s; compute L_K as the mean squared value of (|z|-1) times this singular-value estimate; optimize task_loss+lambda_K L_K. Use A_s in the forward pass. (3) The J-selfadjoint parameterization and inertia factor come from the paper. The spectral radius, sampled resolvent maximum, and finite-horizon power growth max over k less than or equal to T of ||A_s^k||_2 are empirical estimates. For a generic dense H, use r at most 2q+1 rather than estimating d(A). (4) First experiment: compare a width-128 tanh RNN, a spectrally rescaled unconstrained RNN, and this q=4 model on sequential MNIST and the adding or copying task. Match initialization, optimizer, and parameter count as closely as possible. Sweep lambda_K and measure validation accuracy, gradient norm variance, loss after long sequences, sampled K(A), and max over k less than or equal to 500 of ||A_s^k||_2. A successful result is lower transient amplification and fewer exploding gradients with equal or better long-sequence accuracy at comparable training loss and FLOPs. Cache the resolvent penalty every 10 to 50 optimizer steps to reduce overhead.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.