# Lyapunov-gap regularization for recurrent dynamics

- ID: 62
- Canonical URL: https://synthcore.org/idea/62/lyapunov-gap-regularization-for-recurrent-dynamics
- API JSON: https://synthcore.org/api/idea/62.json
- API Markdown: https://synthcore.org/api/idea/62.md
- Verification status: mechanism_failed
- Source: [arXiv:2608.22543](https://arxiv.org/abs/2608.22543)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics
- Math tags: dynamical-systems, random-matrix, spectral-theory, linear-algebra
- Ratings: usefulness 7/10; difficulty 6/10; novelty 6/10

## Idea description

Regularize a recurrent or state-space model using finite-time Lyapunov exponents of its actual hidden-state transition products. Penalize collapsed adjacent exponents while also controlling the largest exponent, encouraging several useful state directions instead of one dominant direction or universal contraction.

## Mathematical statement

For independent matrices M_t in SL_d(R) with E[log ||M_t||] finite, the Furstenberg-Kesten and Kingman result gives deterministic exponents L_1 >= ... >= L_d satisfying L_i = lim_{n to infinity} n^{-1} log sigma_i(M_1 ... M_n) almost surely, where sigma_i(A) is the i-th singular value. The paper proves for its 2W-dimensional transfer matrices that every adjacent gap satisfies L_i - L_{i+1} >= c/W, where c is a positive model-dependent constant. We estimate exponents numerically from QR factors Q_t R_t = M_t Q_{t-1}, with positive diagonal entries R_{t,ii}, using Lhat_i = T^{-1} sum_t log(R_{t,ii}). The regularizer penalizes gaps below delta/W and penalizes Lhat_1 above a chosen stability ceiling rho.

## Key formulas

- $$L_i=\lim_{n\to\infty}\frac{1}{n}\log\sigma_i(M_1\cdots M_n),\qquad L_1\geq\cdots\geq L_d.$$
- $$L_i-L_{i+1}\geq\frac{c}{W},\qquad i=1,\ldots,2W-1.$$
- $$Q_tR_t=M_tQ_{t-1},\qquad \widehat L_i=\frac{1}{T}\sum_{t=1}^{T}\log\left(|R_{t,ii}|+10^{-8}\right).$$
- $$\mathcal L_{\mathrm{gap}}=\sum_{i=1}^{d-1}\left[\max\left(0,\frac{\delta}{W}-(\widehat L_i-\widehat L_{i+1})\right)\right]^2+\beta\left[\max(0,\widehat L_1-\rho)\right]^2.$$

## Implementation notes

(1) Integration point: apply this to the recurrent transition Jacobian, not to the output head. For a linear or diagonal SSM, M_t is the discretized state transition. For a GRU or LSTM, M_t is the Jacobian of h_t with respect to h_{t-1}, evaluated on a minibatch trajectory. Use a smaller number k of leading directions if the hidden dimension is large. (2) Pseudocode: initialize Q with k orthonormal columns; set s=0; for t=1,...,T, compute M_t, form Z=M_t Q, compute reduced QR Z=Q_new R, set Q=Q_new, and accumulate s_i += log(abs(R[i,i])+1e-8); after the loop set Lhat_i=s_i/T and add lambda_gap times the displayed L_gap to the task loss. (3) The QR recursion and exponent estimator come directly from the Lyapunov definition. The target delta/W, ceiling rho, and penalty weights are empirical hyperparameters; estimate c only if desired by measuring gaps in a baseline model. Use stop-gradient through M_t initially for numerical stability, then test differentiable Jacobian products. (4) First experiment: compare a vanilla GRU, orthogonally initialized GRU, spectral-normalized GRU, and this regularizer on sequential MNIST and the adding problem. Record loss versus optimizer steps and FLOPs, gradient norms versus sequence length, hidden-state effective rank, and validation accuracy. Success is fewer exploding or vanishing gradients, larger effective rank, and improved long-context accuracy at equal parameter count and compute.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a tanh-RNN MVP with analytical hidden-state Jacobians, QR-based finite-time Lyapunov exponents, gap penalties, and a stability-ceiling penalty. The core estimator matched both analytic exponents and singular-value product exponents with maximum error about 5.6e-16. In the matched 60-step adding-task run, baseline validation MSE was 0.10809 versus 0.10812 with regularization, while runtime increased from 7.77s to 52.49s; the promised win was not observed.

### Mechanism check

- Verdict: Built a tanh-RNN MVP with analytical hidden-state Jacobians, QR-based finite-time Lyapunov exponents, gap penalties, and a stability-ceiling penalty. The core estimator matched both analytic exponents and singular-value product exponents with maximum error about 5.6e-16. In the matched 60-step adding-task run, baseline validation MSE was 0.10809 versus 0.10812 with regularization, while runtime increased from 7.77s to 52.49s; the promised win was not observed.
- Confidence: 9/10
- Limitations: Only a tiny synthetic adding problem and tanh-RNN were tested; no GRU/LSTM, SSM, sequential MNIST, hyperparameter sweep, multiple seeds, FLOP-normalized comparison, or differentiable-vs-stop-gradient ablation was performed. The current implementation uses full hidden dimension QR and differentiates through the QR penalty, making it substantially more expensive than the baseline.

## Artifacts

- [lyapunov_experiment.py](https://synthcore.org/code/22/lyapunov_experiment.py)
- [report.md](https://synthcore.org/code/22/report.md)
- [results.json](https://synthcore.org/code/22/results.json)
- [Download all files as ZIP](https://synthcore.org/download/22)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.