# Jointly Contractive Input-Conditioned RNN

- ID: 249
- Canonical URL: https://synthcore.org/idea/249/jointly-contractive-input-conditioned-rnn
- API JSON: https://synthcore.org/api/idea/249.json
- API Markdown: https://synthcore.org/api/idea/249.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.03849](https://arxiv.org/abs/2607.03849)
- Category: dynamics
- Solves: stability, generalization, accuracy
- ML areas: rnn, ssm, training-dynamics, initialization
- Math tags: dynamical-systems, linear-algebra, stability, control-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Replace pointwise spectral normalization of an RNN transition with a stability constraint on the entire family of input-conditioned matrices. Use a learned positive-definite metric P so every transition contracts in the same state geometry, approximating the paper's uniform exponential stability and input-forgetting guarantee.

## Mathematical statement

The paper studies state-linear systems x(t+1)=A_{u(t)}x(t), y(t)=Cx(t), where x(t) is the hidden state, u(t) is the input, A_u is the transition matrix selected by the input, and C is the readout. Its central stability principle is that uniform global exponential stability is equivalent to the joint spectral radius of the matrix family {A_u}_{u\in\mathbb{U}} being less than one. The joint spectral radius is defined by rho_jsr({A_u})=limsup_{T    o\infty} sup_{u_0,...,u_{T-1}} ||A_{u_{T-1}}...A_{u_0}||^{1/T}. For an implementable sufficient condition, introduce P positive definite and 0<rho<1 such that A_u^T P A_u is less than or equal to rho^2 P for every admissible u. Then the Lyapunov energy V_t=x_t^T P x_t satisfies V_{t+1}<=rho^2 V_t and therefore ||x_t||_P<=rho^t||x_0||_P, where ||x||_P=sqrt(x^T P x). This common quadratic condition is stronger than joint-spectral-radius stability but is differentiable enough to enforce during neural-network training.

## Key formulas

- $$x(t+1)=A_{u(t)}x(t),\qquad y(t)=Cx(t)$$
- $$\rho_{\mathrm{jsr}}(\{A_u\})=\limsup_{T\to\infty}\ \sup_{u_0,\ldots,u_{T-1}}\left\|A_{u_{T-1}}\cdots A_{u_0}\right\|^{1/T}<1$$
- $$A_u^{\top}PA_u\preceq\rho^2P\quad\forall u\in\mathbb U,\qquad P\succ0,\quad 0<\rho<1$$
- $$V_{t+1}=x(t+1)^\top Px(t+1)\le\rho^2V_t\quad\Longrightarrow\quad\|x(t)\|_P\le\rho^t\|x(0)\|_P$$

## Implementation notes

Integrate this into an input-conditioned GRU-like or state-space recurrent layer whose transition is h_{t+1}=A(z_t)h_t+G(z_t), with z_t being the current token embedding or a low-dimensional projection of it. Start with a finite set of K transition matrices A_k produced by K learned basis matrices and softmax gates: A(z)=sum_k pi_k(z)A_k; enforce the constraint on each basis matrix, which also controls every convex mixture. Parameterize P=L L^T+epsilon I using a lower-triangular L with positive diagonal. At every training step, compute the normalized Lyapunov violation for each basis matrix, r_k=max(0, lambda_max(P^{-1/2}A_k^T P A_k P^{-1/2})-rho^2), and add lambda_L sum_k r_k^2 to the task loss. A cheaper guaranteed variant computes B_k=P^{1/2}A_kP^{-1/2} and parameterizes B_k=rho*tanh(s_k)*Q_k, where Q_k is approximately orthogonal, then maps back to A_k; use the penalty version first because it is less restrictive. Estimate the finite-horizon joint spectral radius empirically by sampling many token sequences and tracking g_T=max_sequences ||A_{T-1}...A_0||_2^{1/T}; this estimate is diagnostic, not a formal guarantee. Pseudocode: initialize h=0; for t in 1..T, compute pi=softmax(router(z_t)), set A=sum_k pi_k*A_k, and update h=A@h+G(z_t). Add loss=task_loss+lambda_L*sum_k*r_k^2, then backpropagate and update A_k and L. Estimate lambda_max with 5 to 10 power iterations rather than full eigendecomposition for large hidden states. The first cheap experiment should compare a 2-layer 256-unit input-conditioned RNN with unconstrained transitions, standard spectral normalization, and the Lyapunov penalty on sequential MNIST and copy/add tasks. Measure gradient norms, hidden-state norms, long-sequence loss, accuracy, and wall-clock overhead at equal parameters and FLOPs. Success means bounded hidden norms, fewer exploding-gradient runs, and better long-context accuracy without materially slowing short-sequence convergence.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built an input-conditioned recurrent layer with a learned positive-definite metric, generalized-eigenvalue Lyapunov penalty, pointwise spectral-normalization control, and an explicit product-contraction sanity check. The core math check passed: all basis matrices satisfied the bound and sampled convex-mixture products contracted substantially in the P-norm. However, on the toy 30-step memory task, the Lyapunov model did not improve over controls: long-sequence accuracy was 0.445 for all variants and final MSE was essentially 1.0, so the promised task-level win was not observed.

### Mechanism check

- Verdict: Built an input-conditioned recurrent layer with a learned positive-definite metric, generalized-eigenvalue Lyapunov penalty, pointwise spectral-normalization control, and an explicit product-contraction sanity check. The core math check passed: all basis matrices satisfied the bound and sampled convex-mixture products contracted substantially in the P-norm. However, on the toy 30-step memory task, the Lyapunov model did not improve over controls: long-sequence accuracy was 0.445 for all variants and final MSE was essentially 1.0, so the promised task-level win was not observed.
- Confidence: 8/10
- Limitations: Only a small synthetic one-bit memory task was tested, with an 8-unit single-layer recurrent model rather than the proposed larger GRU-like architecture or sequential MNIST/copy-add benchmarks. The task setup and rho=0.90 constraint may make 30-step memory intrinsically difficult; no hyperparameter sweep, formal JSR computation, or repeated-seed statistical comparison was performed.

## Artifacts

- [experiment.py](https://synthcore.org/code/1241/experiment.py)
- [report.md](https://synthcore.org/code/1241/report.md)
- [Download all files as ZIP](https://synthcore.org/download/1241)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.