Lyapunov-Continuation Initialization for Noisy State-Space Layers

(1) Exact integration point: apply the method to an explicit linear state-space or gated RNN layer. Parameterize the hidden transition as $h_{t+1}=h_t+\Delta t[(A-BK)h_t+Bu_t]+\sqrt{\Delta t\rho}(C-DK)h_t\epsilon_t$, with $\epsilon_t\sim\mathcal{N}(0,1)$. The matrices $A,B,C,D$ can be fixed small random matrices or learned matrices, while $K$ and input/output projections are trainable. (2) Pseudocode: initialize $\rho=0$; parameterize $P=LL^\top+\varepsilon I$; at every continuation stage compute $A_K=A-BK$, $C_K=C-DK$, and residual $R=A_K^\top P+PA_K+\rho C_K^\top PC_K+\alpha P$. Optimize task loss plus $\lambda\max(0,\lambda_{\max}(R))^2$, or use $\lambda\|[R]_+\|_F^2$ where $[R]_+$ retains positive eigenvalues. Increase $\rho$ from $0$ to $1$ over $J$ stages, warm-starting $K,L$ at each stage. During each recurrent step apply the stochastic update above. (3) Compute from the paper's mathematics the Lyapunov residual and its largest eigenvalue. For large state dimension, estimate $\lambda_{\max}(R)$ with 5--10 power iterations; empirically record hidden-state RMS, gradient norm, and the fraction of steps violating the certificate. The theorem does not directly cover nonlinear network components, so their behavior must be measured. (4) First cheap experiment: compare a 2-layer 256-dimensional noisy linear SSM/RNN on sequential MNIST or permuted MNIST against the same model with orthogonal or Xavier initialization and no Lyapunov penalty. Match parameter count, optimizer, batch size, and compute. Evaluate at multiplicative-noise amplitudes $0$, $0.25$, $0.5$, and $1.0$. Success criteria are fewer exploding-state runs, lower maximum hidden-state RMS, bounded gradient norms, and equal or better validation accuracy at matched FLOPs. A strong positive signal is faster loss descent after increasing $\rho$ and graceful accuracy degradation under stronger noise.