# Full-block IQC certificates for stable RNNs

- ID: 1323
- Canonical URL: https://synthcore.org/idea/1323/full-block-iqc-certificates-for-stable-rnns
- API JSON: https://synthcore.org/api/idea/1323.json
- API Markdown: https://synthcore.org/api/idea/1323.md
- Verification status: unverified
- Source: [arXiv:2607.23599](https://arxiv.org/abs/2607.23599)
- Category: dynamics
- Solves: stability, generalization
- ML areas: rnn, training-dynamics
- Math tags: control-theory, dynamical-systems, convex-analysis, linear-algebra
- Ratings: usefulness 6/10; difficulty 7/10; novelty 6/10

## Idea description

Model an RNN as a linear state-space system in feedback with its slope-restricted activation, then search for a finite-horizon IQC multiplier instead of relying only on a spectral-radius or OZF-style condition. Penalize or reject parameter settings for which the strict IQC/LMI certificate has insufficient margin, yielding a directly testable stability criterion for long unrolled sequences.

## Mathematical statement

The paper assumes a memoryless nonlinearity \(\phi\) satisfying the slope restriction \(0\leq(\phi(x_1)-\phi(x_2))/(x_1-x_2)\leq\kappa\), where \(\kappa\) is the activation slope bound. For an RNN, let \(G\) be the stable linear state-space system with state \(h_t\), input \(u_t\), output \(y_t\), and let \(\Delta\) represent the activation feedback. A lifting filter \(\Psi\) maps the input-output sequence to \(z_t\), and \(\Pi\in\mathbb{S}^{2N\times 2N}\) is a multiplier such that every admissible \(\Delta\) satisfies the hard IQC \(\sum_{t=0}^{T}z_t^\top\Pi z_t\geq0\) for every finite horizon \(T\). Theorem 3 states that stability follows if there is a symmetric matrix \(\mathcal{X}\in\mathbb{S}^{(n_\Psi+n)\times(n_\Psi+n)}\) with \(\mathcal{X}\succeq0\), \(\operatorname{tr}(\mathcal{X})=1\), and \(\begin{pmatrix}I&0\\\mathcal{A}&\mathcal{B}\\\mathcal{C}&\mathcal{D}\end{pmatrix}^{\!\top}\begin{pmatrix}-\mathcal{X}&0&0\\0&\mathcal{X}&0\\0&0&\Pi\end{pmatrix}\begin{pmatrix}I&0\\\mathcal{A}&\mathcal{B}\\\mathcal{C}&\mathcal{D}\end{pmatrix}\prec0\). Here \(n\) is the RNN state dimension, \(n_\Psi\) is the lifting-filter state dimension, and \((\mathcal{A},\mathcal{B},\mathcal{C},\mathcal{D})\) is the augmented realization of the RNN linear dynamics cascaded with \(\Psi\). The strict negative-definiteness margin is the certificate quantity to maximize.

## Key formulas

- $$0\leq\frac{\phi(x_{1})-\phi(x_{2})}{x_{1}-x_{2}}\leq\kappa\qquad\text{for all }x_1,x_2\in\mathbb{R}.$$
- $$\sum_{t=0}^{T} z_t^{\top}\Pi z_t\geq 0,\qquad z=\Psi\begin{bmatrix}u\\y\end{bmatrix}.$$
- $$\begin{pmatrix}I&0\\\mathcal{A}&\mathcal{B}\\\mathcal{C}&\mathcal{D}\end{pmatrix}^{\!\top}\begin{pmatrix}-\mathcal{X}&0&0\\0&\mathcal{X}&0\\0&0&\Pi\end{pmatrix}\begin{pmatrix}I&0\\\mathcal{A}&\mathcal{B}\\\mathcal{C}&\mathcal{D}\end{pmatrix}\prec0,\qquad \mathcal{X}\succeq0,\quad \operatorname{tr}(\mathcal{X})=1.$$
- $$h_{t+1}=Ah_t+Bu_t+B_\phi\phi(C_\phi h_t+D_\phi u_t),\qquad y_t=C_yh_t+D_yu_t.$$
- $$M(\mathcal X,\Pi)+\rho I\preceq0,\qquad \mathcal L_{\mathrm{IQC}}=\lambda\max(0,\rho_0-\rho)^2.$$

## Implementation notes

1. Integration point: apply this to a tanh or ReLU RNN, or to a residual sequence block, during architecture search or periodically during training. Write the cell as \(h_{t+1}=Ah_t+Bu_t+B_\phi v_t\), \(v_t=\phi(C_\phi h_t+D_\phi u_t)\), with output \(y_t=C_yh_t+D_yu_t\). For tanh use \(\kappa=1\); for ReLU use the sector bound \(0\leq\phi'\leq1\). 2. Pseudocode: construct the augmented state-space matrices \((\mathcal A,\mathcal B,\mathcal C,\mathcal D)\) from the RNN matrices and a short FIR lifting filter \(\Psi\) with memory \(N\). Choose a sector-valid symmetric multiplier \(\Pi\), solve an SDP for \(\mathcal X\succeq0\) and \(\operatorname{tr}(\mathcal X)=1\), and maximize \(\rho\) subject to \(M(\mathcal X,\Pi)+\rho I\preceq0\), where \(M\) is the left-hand side of the IQC inequality. Add \(\lambda\max(0,\rho_0-\rho)^2\) to the network loss, or reject optimizer updates with \(\rho<\rho_0\). 3. Compute the augmented matrices and LMI exactly; estimate only current Jacobian blocks if the architecture is nonlinear in its parameters. For an MVP, keep \(\Pi\) fixed to a static sector multiplier, then compare it against an optimized full-block/FIR multiplier. 4. Test a two-layer tanh RNN on adding and permuted sequential MNIST, comparing spectral normalization and gradient clipping. Track \(\rho\), exploding-gradient frequency, hidden-state norms over 1,000-step rollouts, validation accuracy, and SDP overhead. Success means fewer unstable long rollouts and improved long-horizon accuracy at matched parameter count and comparable training cost.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.