# Geometrically Attracting Random Recurrent Layer

- ID: 2803
- Canonical URL: https://synthcore.org/idea/2803/geometrically-attracting-random-recurrent-layer
- API JSON: https://synthcore.org/api/idea/2803.json
- API Markdown: https://synthcore.org/api/idea/2803.md
- Verification status: unverified
- Source: [arXiv:2608.29022](https://arxiv.org/abs/2608.29022)
- Category: dynamics
- Solves: stability, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: dynamical-systems, probability, control-theory, linear-algebra
- Ratings: usefulness 7/10; difficulty 5/10; novelty 6/10

## Idea description

Replace a recurrent update by a time-inhomogeneous random choice among candidate maps, and regulate the candidate Jacobian gains so that the expected product of gains contracts geometrically. This should make hidden-state distributions forget their initial state even when the map family and selection probabilities vary over time, improving long-horizon stability without requiring every individual candidate map to be strongly contractive.

## Mathematical statement

Let Xₜ₊₁ = Fₜ,ᴵₜ(Xₜ), where Fₜ,ᵢ is candidate map i at time t and Iₜ is sampled with probability pₜ,ᵢ. Let Lₜ,ᵢ satisfy ||Fₜ,ᵢ(x) − Fₜ,ᵢ(y)|| ≤ Lₜ,ᵢ ||x − y||. A sufficient geometric-attraction condition is E[Πₜ₌₀ⁿ⁻¹ Lₜ,ᴵₜ] ≤ Cρⁿ for constants C < infinity and 0 < ρ < 1. For independent selections this reduces to Πₜ₌₀ⁿ⁻¹ Σᵢ pₜ,ᵢLₜ,ᵢ ≤ Cρⁿ. If μₙ and νₙ are the state distributions generated from two initial distributions, their bounded-Lipschitz distance should decay geometrically: d_BL(μₙ,νₙ) ≤ Cρⁿd_BL(μ₀,ν₀). In the neural-network implementation, Lₜ,ᵢ is estimated by the spectral norm of the candidate hidden-state Jacobian, and the logarithmic average gain is regularized below a target log contraction rate.

## Key formulas

- $$X_{t+1}=F_{t,I_t}(X_t),\quad P(I_t=i)=p_{t,i},\quad ||F_{t,i}(x)-F_{t,i}(y)||\le L_{t,i}||x-y||.$$
- $$E[\prod_{t=0}^{n-1}L_{t,I_t}]\le C\rho^n,\quad 0<\rho<1;\qquad \prod_{t=0}^{n-1}\left(\sum_i p_{t,i}L_{t,i}\right)\le C\rho^n.$$
- $$d_{BL}(\mu_n,\nu_n)\le C\rho^n d_{BL}(\mu_0,\nu_0),\quad d_{BL}(\mu,\nu)=\sup_{||f||_\infty\le1,\operatorname{Lip}(f)\le1}|E_\mu f-E_\nu f|.$$
- $$L_{contract}=\frac{1}{T}\sum_{t=0}^{T-1}\operatorname{softplus}\left(\log\left(\sum_i p_{t,i}\widehat L_{t,i}\right)-\log\rho_*\right)^2.$$

## Implementation notes

Integrate the mechanism into an RNN or state-space block with K candidate maps, such as Fₜ,ᵢ(hₜ,xₜ) = tanh(Wₜ,ᵢhₜ + Uₜ,ᵢxₜ + bₜ,ᵢ), and a time-dependent categorical gate pₜ,ᵢ. At each training step, estimate each candidate gain L̂ₜ,ᵢ using one to three power-iteration steps on the hidden-to-hidden Jacobian. A cheaper upper bound is ||Wₜ,ᵢ||₂ because tanh is 1-Lipschitz. Pseudocode: compute pₜ; sample a candidate or use the probability-weighted mixture; update hₜ₊₁ = Fₜ,ᴵₜ(hₜ,xₜ); estimate all L̂ₜ,ᵢ; compute gₜ = log(Σᵢ pₜ,ᵢL̂ₜ,ᵢ + epsilon); add λ times the squared positive part of gₜ − log(ρ*) to the task loss. Track γ̂ₜ = T⁻¹Σₜgₜ, and optionally rescale recurrent matrices when γ̂ₜ exceeds zero. The paper supplies the contraction mechanism; Jacobian estimates, routing probabilities, and finite-horizon exponents are measured empirically. First test on sequential MNIST and the adding problem against a GRU and an unconstrained random-map RNN with matched parameter counts. Run two identical input sequences from hidden states separated by unit norm, and record dₜ = ||hₜ − h′ₜ|| for 200 steps. The prediction is a stability boundary near γ̂ = 0: below it, log dₜ has negative slope approximately γ̂; above it, distances do not decay geometrically. The measured slope should agree with the accumulated log-gain within about 20 percent.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.