Idea description

Constrain a recurrent transition so that its dynamically relevant invariant subspaces have no eigenvalues near the unit circle, separating contracting memory directions from expanding prediction directions. Add a pseudo-orbit consistency loss so that trajectories generated with bounded transition perturbations remain close to clean trajectories, as expected from hyperbolic shadowing.

Formulas

$$\|y_{t+1}-F(y_t)\|\leq\delta,\qquad x_{t+1}=F(x_t),\qquad \sup_t\|x_t-y_t\|\leq\varepsilon(\delta),\qquad \varepsilon(\delta)\to0.$$

$$\operatorname{spec}(T_{\mathbb Q}|_W)\cap\{z\in\mathbb C:|z|=1\}=\varnothing\quad\text{for every finite-dimensional invariant subspace }W.$$

$$R_{\mathrm{hyp}}(\theta)=\frac{1}{T}\sum_{t=0}^{T-1}\sum_{j=1}^{k}\max\left(0,\mu^2-(|\lambda_{t,j}|-1)^2\right),\qquad J_t=\frac{\partial F_\theta}{\partial h}(h_t,u_t).$$

$$\mathcal L=\mathcal L_{\mathrm{task}}+\rho R_{\mathrm{hyp}}+\kappa\frac{1}{T}\sum_{t=0}^{T-1}\|\widetilde h_{t+1}-F_\theta(\widetilde h_t,u_t)\|^2,\qquad \widetilde h_{t+1}=F_\theta(\widetilde h_t,u_t)+\xi_t,\quad \|\xi_t\|\leq\delta.$$

Mathematical statement

For a discrete map F and tolerance delta greater than zero, a sequence y_t is a delta-pseudo-orbit when the one-step defect satisfies ||y_(t+1) - F(y_t)|| <= delta. Shadowing means that there is an exact orbit x_(t+1) = F(x_t) with sup_t ||x_t - y_t|| <= epsilon(delta), where epsilon(delta) tends to zero as delta tends to zero. The paper's linear criterion requires every finite-dimensional invariant subspace W of the rational linear operator T_Q to be hyperbolic: its spectrum must not intersect the unit circle. For a neural transition h_(t+1) = F_theta(h_t,u_t), estimate the Jacobian J_t = partial F_theta / partial h and penalize eigenvalues close to modulus one. The target is a spectral gap gamma: every monitored eigenvalue should satisfy |lambda| <= 1 - gamma or |lambda| >= 1 + gamma.

Implementation notes

Integrate this into a residual RNN or diagonal state-space model with hidden transition h_(t+1) = F_theta(h_t,u_t). For the first implementation, use F_theta(h,u) = A h + B u + g_theta(h,u), with A diagonal or block diagonal, so transition eigenvalues are cheap to compute. Run a clean sequence to obtain h_t. Every m steps, estimate the largest few Jacobian eigenvalues using Jacobian-vector products and Arnoldi iteration, or compute them exactly for the diagonal A model. Add R_hyp to the task loss. In a second forward pass, inject bounded noise xi_t into each transition and minimize the resulting pseudo-orbit consistency loss against the clean hidden states. The paper supplies the pseudo-orbit and hyperbolicity mechanism; finite-network spectra, the spectral gap, and the shadowing constant are empirical estimates. First experiment: compare a vanilla tanh RNN, spectral-normalized RNN, and this model on sequential MNIST and the adding problem with sequence lengths 100, 500, and 1000. Sweep the recurrent spectral radius from 0.8 to 1.2 and inject noise with delta equal to 10^-4, 10^-3, and 10^-2. Measure hidden-state deviation versus delta and the distance of eigenvalues from the unit circle. The quantitative prediction is that stable shadowing error scales approximately as delta divided by gamma, while entering the interval [1-gamma,1+gamma] causes a sharp increase in deviation. The measured divergence boundary should agree with the estimated unit-circle crossing within 20 percent.

Verification

Failed on benchmark

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Not reproduced ✗

Stage 2 · Practical benchmark result: Significantly worse ✗

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a Hyperbolic Shadowing RNN MVP with diagonal recurrent dynamics, Jacobian spectral-gap regularization, bounded pseudo-orbit noise, and consistency loss. The toy verification confirmed the predicted delta/gamma shadowing scaling and sharp growth at the unit-circle boundary. The adding-task comparison showed no accuracy win: the hyperbolic model was slightly worse than the vanilla RNN, so the mechanism was demonstrated but the task-level benefit was not.

Agent confidence: 9/10

Baseline: Adding-task test MSE 0.1813023 after 500 Adam steps
Idea: Adding-task test MSE 0.1815263; stable-bound ratio mean 0.99999996; inverse-gap normalized scaling approximately 1.0

Limitations: Only a scalar linear sweep and length-100 adding task were tested. Sequential MNIST, lengths 500 and 1000, spectral-normalized RNN comparison, stochastic-noise statistics, full Jacobian/Arnoldi spectra, and broader hyperparameter sweeps were not tested.

How to run: python3 experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2608.00955) Sep 2, 2026

Significantly WORSE than the tuned baseline (+2915.3%, p=0.0081, wins 0 / 8 paired seeds).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0032
Idea mean: 0.0956
Effect (Δ): 0.0924 (+2915.3%; negative = idea better)
Wins: 0 / 8 paired seeds
p-value: 0.0081 (permutation test, 20 000 shuffles)
Smallest detectable effect: n/a
Mechanism: Not confirmed ✗
Practical effect: Hurts
Baseline tuning: swept over 6 configs

Limitations:

Tested only the built-in dynamics track with 400 training samples, 400 test samples, 10 epochs, and the shared rnn_small GRU. The intervention monitored recurrent GRU gate-matrix spectra rather than full time-varying nonlinear Jacobian spectra; longer horizons, Arnoldi/full-Jacobian estimates, broader noise statistics, and a separately tuned spectral-normalized baseline were not tested.

How to run: python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 31, 2026 latest Failed ✗

Agent confidence: 10/10

Built and ran the full eight-seed Stage-2 dynamics benchmark using the matched rnn_small GRU architecture. The hyperbolic intervention was significantly worse than the tuned baseline (paired delta +0.09244937 MSE, permutation p=0.0081), and the trained-model shadowing signature did not confirm the predicted delta/gap scaling. The idea therefore did not transfer as a task-level win under this protocol.

Baseline: dynamics/rnn_small; tuned lr=0.005, weight_decay=0.0001; full 8-seed test MSE mean 0.00317115, std 0.00111026
Idea: best of 3 settings at lr=0.005, weight_decay=0.0001, rho=0.01, kappa=0.05; full 8-seed test MSE mean 0.09562052, std 0.02706381; paired delta +0.09244937, p=0.0081

Benchmark result