# Lyapunov Canonical-Angle Regularizer

- ID: 132
- Canonical URL: https://synthcore.org/idea/132/lyapunov-canonical-angle-regularizer
- API JSON: https://synthcore.org/api/idea/132.json
- API Markdown: https://synthcore.org/api/idea/132.md
- Verification status: unverified
- Source: [arXiv:2608.24405](https://arxiv.org/abs/2608.24405)
- Category: regularization
- Solves: stability, generalization
- ML areas: ssm, rnn, regularization
- Math tags: linear-algebra, dynamical-systems, control-theory, spectral-theory
- Ratings: usefulness 5/10; difficulty 5/10; novelty 6/10

## Idea description

Add a spectral regularizer to a linear state-space or recurrent layer that controls the overlap between its controllable and observable state directions. The regularizer uses the paper's identity to monitor eigenvalues of (I+PQ)^{-1}, equivalently the squared canonical correlations between reachable and observable subspaces, and penalizes degenerate or overly concentrated spectra.

## Mathematical statement

For a stable linear recurrent system x_{t+1}=Ax_t+Bu_t and y_t=Cx_t, define the infinite controllability matrix C_inf=[B,AB,A^2B,...] and observability matrix O_inf=[C^T,A^T C^T,(A^T)^2 C^T,...]. The Gramians are P=C_inf C_inf^T and Q=O_inf O_inf^T, while R=C_inf O_inf^T. The paper proves that P^{-1} R Q^{-1} R^T and (I+PQ)^{-1} have the same characteristic polynomial for the coupled Lyapunov/Sylvester construction associated with the rank-one update A_2=A+vw^T. In the Schur-stable realization, the eigenvalues of the first matrix are squared canonical correlations cos^2(theta_i) between range(C_inf^T) and range(O_inf^T), and equivalently the eigenvalues of PQ are tan^2(theta_i). For implementation, approximate the infinite matrices with a finite horizon T, add ridge epsilon I to P and Q, and regularize the eigenvalues lambda_i=(1+s_i)^{-1}, where s_i are eigenvalues of the symmetric matrix Q^{1/2} P Q^{1/2}.

## Key formulas

- $$P=\mathcal{C}_{\infty}\mathcal{C}_{\infty}^{\top},\qquad Q=\mathcal{O}_{\infty}\mathcal{O}_{\infty}^{\top},\qquad R=\mathcal{C}_{\infty}\mathcal{O}_{\infty}^{\top}.$$
- $$P^{-1}RQ^{-1}R^{\top}\quad\text{and}\quad (I+PQ)^{-1}\quad\text{have the same characteristic polynomial}.$$
- $$\lambda_i\!\left((I+PQ)^{-1}\right)=\cos^2\theta_i,\qquad \lambda_i(PQ)=\tan^2\theta_i.$$
- $$\mathcal{L}_{\mathrm{angle}}=\frac{1}{n}\sum_{i=1}^{n}\left[\log(\lambda_i+\delta)-\log(\lambda_{\mathrm{target}}+\delta)\right]^2.$$

## Implementation notes

Integrate this into the recurrent or state-space block, rather than into the output head. Let the hidden dimension be n, with transition A, input projection B, and readout C. First stabilize A, for example with A=rho A_raw/(max(1,||A_raw||_2)+1e-4), where rho is a learned or fixed scalar constrained below one. Every K optimizer steps, construct finite-horizon Gramians using T unrolled powers: initialize P=0, Q=0, Xc=B, Xo=C^T; for k=0,...,T-1, update P=P+Xc Xc^T, Q=Q+Xo Xo^T, then Xc=A Xc and Xo=A^T Xo. Replace P and Q by P+epsilon I and Q+epsilon I, with epsilon=1e-4 trace(P)/n. Compute Q^{1/2} by eigendecomposition, form S=Q^{1/2} P Q^{1/2}, and compute its eigenvalues s_i. Set lambda_i=1/(1+s_i), then add alpha times L_angle to the task loss. Use lambda_target=0.5 initially, or instead penalize spectral variance. Detach the eigensystem every K steps if memory is limiting. The theorem supplies the canonical-angle interpretation and the PQ to (I+PQ)^{-1} conversion; finite horizon, ridge, and target spectrum are empirical choices. First test a 256-unit Elman RNN and a small diagonal-plus-low-rank SSM on sequential MNIST, comparing against weight decay alone. Track validation accuracy, gradient norms over sequence time, Gramian condition numbers, and failures from exploding gradients. A positive result is equal or higher accuracy with fewer unstable runs and improved extrapolation from sequence length 100 to 500.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.