# Uniformly Mixing Nonstationary State-Space Network

- ID: 2808
- Canonical URL: https://synthcore.org/idea/2808/uniformly-mixing-nonstationary-state-space-network
- API JSON: https://synthcore.org/api/idea/2808.json
- API Markdown: https://synthcore.org/api/idea/2808.md
- Verification status: unverified
- Source: [arXiv:2608.29135](https://arxiv.org/abs/2608.29135)
- Category: dynamics
- Solves: stability, generalization
- ML areas: rnn, ssm, training-dynamics, regularization
- Math tags: dynamical-systems, probability, statistics, control-theory
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Build a recurrent or state-space network with time-dependent transition parameters, but train it to forget perturbations at a common exponential rate across all admissible parameter schedules. The model should retain task-relevant long-term signals while suppressing dependence on arbitrary initial hidden states, reducing instability under changing inputs, curricula, or deployment-time dynamics.

## Mathematical statement

The paper proves that for a nonstationary composition $T_{1,n}=T_n\circ\cdots\circ T_1$ of admissible heterochaos maps, two absolutely continuous initial measures $\mu$ and $\nu$ with Hölder densities satisfy $\|(T_{1,n})_*\mu-(T_{1,n})_*\nu\|_{\mathrm{TV}}\le C s^n$ for constants $C>0$ and $s\in(0,1)$ independent of the admissible parameter sequence. Here $(T_{1,n})_*$ is pushforward and $\|\cdot\|_{\mathrm{TV}}$ is total variation distance. Adapt this as a distributional forgetting condition for hidden states $h_{t+1}=F_{\theta_t}(h_t,x_t)$: for two initial hidden-state distributions $P_0,Q_0$ exposed to the same input sequence, require $D(P_n,Q_n)\le C\rho^n$, where $D$ is an empirical Wasserstein-1 or random-projection distance and $\rho\in(0,1)$. The useful property is a schedule-independent decay envelope, not merely contraction for one fixed transition.

## Key formulas

- \left\|(T_{1,n})_*\mu-(T_{1,n})_*\nu\right\|_{\mathrm{TV}}\le C s^n,\qquad T_{1,n}=T_n\circ\cdots\circ T_1,\quad 0<s<1.
- h_{t+1}=F_{\theta_t}(h_t,x_t),\qquad D(P_n,Q_n)\le C\rho^n,\quad 0<\rho<1.
- \mathcal{L}_{\mathrm{mix}}=\sum_{k=1}^{K}\left[\max\left(0,\log D_k-\log C+k\log\rho\right)\right]^2.
- \widehat\rho=\exp\left(\frac{1}{b-a}\sum_{k=a+1}^{b}\log\frac{\widehat D_k}{\widehat D_{k-1}}\right),\qquad \widehat D_k=D(P_k,Q_k).

## Implementation notes

Integrate this into a GRU, diagonal SSM, or small residual RNN by making the transition parameters $\theta_t$ depend on the input, layer, training step, or an externally specified schedule. For every minibatch, duplicate each sequence with two hidden-state initializations: $h_0^{(1)}$ and $h_0^{(2)}$, sampled from a broad fixed distribution or set to two deterministic points. Run both copies with identical inputs and compute $\widehat D_k$ after every $k$-step block. A cheap choice is the average absolute difference across hidden coordinates; a stronger choice is Wasserstein-1 estimated by one-dimensional random projections. Add $\mathcal L_{\mathrm{mix}}$ above, using target envelope $C\rho^k$, and optionally clip or reject transition updates that increase the estimated slope. Pseudocode: initialize two states; for $t=1,\ldots,L$, compute both recurrent updates with the same $x_t$; every $k$ steps compute $D_k$; accumulate the hinge penalty; backpropagate task loss plus $\lambda\mathcal L_{\mathrm{mix}}$. The paper supplies the mechanism and exponential form, but not neural-network constants $C$ or $\rho$; estimate them from a held-out set of schedules by fitting the slope of $\log \widehat D_k$. First experiment: compare a GRU baseline and the constrained GRU on sequential MNIST or permuted sequential MNIST under random time-varying input-gate biases. Use hidden-state initializations separated by a fixed norm, measure $\widehat D_k$ for 100 steps, and test schedules not seen during training. The quantitative prediction is a near-linear plot of $\log \widehat D_k$ versus $k$ with slope $\log\rho<0$, and similar slopes across schedules; fitted slopes should vary by less than 20% and the distance should fall below $C\rho^k$ after the penalty is active. An ablation without the mixing loss should show schedule-dependent decay or occasional growth.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.