# Hyperbolic Shadowing RNN

- ID: 1648
- Canonical URL: https://synthcore.org/idea/1648/hyperbolic-shadowing-rnn
- API JSON: https://synthcore.org/api/idea/1648.json
- API Markdown: https://synthcore.org/api/idea/1648.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.00955](https://arxiv.org/abs/2608.00955)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: dynamical-systems, control-theory, spectral-theory, linear-algebra
- Ratings: usefulness 7/10; difficulty 6/10; novelty 6/10

## Idea description

Constrain a recurrent transition so that its dynamically relevant invariant subspaces have no eigenvalues near the unit circle, separating contracting memory directions from expanding prediction directions. Add a pseudo-orbit consistency loss so that trajectories generated with bounded transition perturbations remain close to clean trajectories, as expected from hyperbolic shadowing.

## Mathematical statement

For a discrete map F and tolerance delta greater than zero, a sequence y_t is a delta-pseudo-orbit when the one-step defect satisfies ||y_(t+1) - F(y_t)|| <= delta. Shadowing means that there is an exact orbit x_(t+1) = F(x_t) with sup_t ||x_t - y_t|| <= epsilon(delta), where epsilon(delta) tends to zero as delta tends to zero. The paper's linear criterion requires every finite-dimensional invariant subspace W of the rational linear operator T_Q to be hyperbolic: its spectrum must not intersect the unit circle. For a neural transition h_(t+1) = F_theta(h_t,u_t), estimate the Jacobian J_t = partial F_theta / partial h and penalize eigenvalues close to modulus one. The target is a spectral gap gamma: every monitored eigenvalue should satisfy |lambda| <= 1 - gamma or |lambda| >= 1 + gamma.

## Key formulas

- $$\|y_{t+1}-F(y_t)\|\leq\delta,\qquad x_{t+1}=F(x_t),\qquad \sup_t\|x_t-y_t\|\leq\varepsilon(\delta),\qquad \varepsilon(\delta)\to0.$$
- $$\operatorname{spec}(T_{\mathbb Q}|_W)\cap\{z\in\mathbb C:|z|=1\}=\varnothing\quad\text{for every finite-dimensional invariant subspace }W.$$
- $$R_{\mathrm{hyp}}(\theta)=\frac{1}{T}\sum_{t=0}^{T-1}\sum_{j=1}^{k}\max\left(0,\mu^2-(|\lambda_{t,j}|-1)^2\right),\qquad J_t=\frac{\partial F_\theta}{\partial h}(h_t,u_t).$$
- $$\mathcal L=\mathcal L_{\mathrm{task}}+\rho R_{\mathrm{hyp}}+\kappa\frac{1}{T}\sum_{t=0}^{T-1}\|\widetilde h_{t+1}-F_\theta(\widetilde h_t,u_t)\|^2,\qquad \widetilde h_{t+1}=F_\theta(\widetilde h_t,u_t)+\xi_t,\quad \|\xi_t\|\leq\delta.$$

## Implementation notes

Integrate this into a residual RNN or diagonal state-space model with hidden transition h_(t+1) = F_theta(h_t,u_t). For the first implementation, use F_theta(h,u) = A h + B u + g_theta(h,u), with A diagonal or block diagonal, so transition eigenvalues are cheap to compute. Run a clean sequence to obtain h_t. Every m steps, estimate the largest few Jacobian eigenvalues using Jacobian-vector products and Arnoldi iteration, or compute them exactly for the diagonal A model. Add R_hyp to the task loss. In a second forward pass, inject bounded noise xi_t into each transition and minimize the resulting pseudo-orbit consistency loss against the clean hidden states. The paper supplies the pseudo-orbit and hyperbolicity mechanism; finite-network spectra, the spectral gap, and the shadowing constant are empirical estimates. First experiment: compare a vanilla tanh RNN, spectral-normalized RNN, and this model on sequential MNIST and the adding problem with sequence lengths 100, 500, and 1000. Sweep the recurrent spectral radius from 0.8 to 1.2 and inject noise with delta equal to 10^-4, 10^-3, and 10^-2. Measure hidden-state deviation versus delta and the distance of eigenvalues from the unit circle. The quantitative prediction is that stable shadowing error scales approximately as delta divided by gamma, while entering the interval [1-gamma,1+gamma] causes a sharp increase in deviation. The measured divergence boundary should agree with the estimated unit-circle crossing within 20 percent.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: harms
- Verdict: Built a Hyperbolic Shadowing RNN MVP with diagonal recurrent dynamics, Jacobian spectral-gap regularization, bounded pseudo-orbit noise, and consistency loss. The toy verification confirmed the predicted delta/gamma shadowing scaling and sharp growth at the unit-circle boundary. The adding-task comparison showed no accuracy win: the hyperbolic model was slightly worse than the vanilla RNN, so the mechanism was demonstrated but the task-level benefit was not.

### Mechanism check

- Verdict: Built a Hyperbolic Shadowing RNN MVP with diagonal recurrent dynamics, Jacobian spectral-gap regularization, bounded pseudo-orbit noise, and consistency loss. The toy verification confirmed the predicted delta/gamma shadowing scaling and sharp growth at the unit-circle boundary. The adding-task comparison showed no accuracy win: the hyperbolic model was slightly worse than the vanilla RNN, so the mechanism was demonstrated but the task-level benefit was not.
- Confidence: 9/10
- Limitations: Only a scalar linear sweep and length-100 adding task were tested. Sequential MNIST, lengths 500 and 1000, spectral-normalized RNN comparison, stochastic-noise statistics, full Jacobian/Arnoldi spectra, and broader hyperparameter sweeps were not tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0081
- Paired wins: 0/8
- Benchmark verdict: idea worse (significant)

## Artifacts

- [REPORT.md](https://synthcore.org/code/612/REPORT.md)
- [bench_report.json](https://synthcore.org/code/612/bench_report.json)
- [bench_run.py](https://synthcore.org/code/612/bench_run.py)
- [experiment.py](https://synthcore.org/code/612/experiment.py)
- [report.md](https://synthcore.org/code/612/report.md)
- [report_bench_2026-08-31T195208.md](https://synthcore.org/code/612/report_bench_2026-08-31T195208.md)
- [results.json](https://synthcore.org/code/612/results.json)
- [Download all files as ZIP](https://synthcore.org/download/612)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.