Uniformly Mixing Nonstationary State-Space Network
Source paper: Decay of correlations and normal approximation for nonstationary heterochaos baker maps arXiv:2608.29135 ⓘ · analyzed Sep 1, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Build a recurrent or state-space network with time-dependent transition parameters, but train it to forget perturbations at a common exponential rate across all admissible parameter schedules. The model should retain task-relevant long-term signals while suppressing dependence on arbitrary initial hidden states, reducing instability under changing inputs, curricula, or deployment-time dynamics.
Formulas
Mathematical statement
The paper proves that for a nonstationary composition $T_{1,n}=T_n\circ\cdots\circ T_1$ of admissible heterochaos maps, two absolutely continuous initial measures $\mu$ and $\nu$ with Hölder densities satisfy $\|(T_{1,n})_*\mu-(T_{1,n})_*\nu\|_{\mathrm{TV}}\le C s^n$ for constants $C>0$ and $s\in(0,1)$ independent of the admissible parameter sequence. Here $(T_{1,n})_*$ is pushforward and $\|\cdot\|_{\mathrm{TV}}$ is total variation distance. Adapt this as a distributional forgetting condition for hidden states $h_{t+1}=F_{\theta_t}(h_t,x_t)$: for two initial hidden-state distributions $P_0,Q_0$ exposed to the same input sequence, require $D(P_n,Q_n)\le C\rho^n$, where $D$ is an empirical Wasserstein-1 or random-projection distance and $\rho\in(0,1)$. The useful property is a schedule-independent decay envelope, not merely contraction for one fixed transition.
Implementation notes
Integrate this into a GRU, diagonal SSM, or small residual RNN by making the transition parameters $\theta_t$ depend on the input, layer, training step, or an externally specified schedule. For every minibatch, duplicate each sequence with two hidden-state initializations: $h_0^{(1)}$ and $h_0^{(2)}$, sampled from a broad fixed distribution or set to two deterministic points. Run both copies with identical inputs and compute $\widehat D_k$ after every $k$-step block. A cheap choice is the average absolute difference across hidden coordinates; a stronger choice is Wasserstein-1 estimated by one-dimensional random projections. Add $\mathcal L_{\mathrm{mix}}$ above, using target envelope $C\rho^k$, and optionally clip or reject transition updates that increase the estimated slope. Pseudocode: initialize two states; for $t=1,\ldots,L$, compute both recurrent updates with the same $x_t$; every $k$ steps compute $D_k$; accumulate the hinge penalty; backpropagate task loss plus $\lambda\mathcal L_{\mathrm{mix}}$. The paper supplies the mechanism and exponential form, but not neural-network constants $C$ or $\rho$; estimate them from a held-out set of schedules by fitting the slope of $\log \widehat D_k$. First experiment: compare a GRU baseline and the constrained GRU on sequential MNIST or permuted sequential MNIST under random time-varying input-gate biases. Use hidden-state initializations separated by a fixed norm, measure $\widehat D_k$ for 100 steps, and test schedules not seen during training. The quantitative prediction is a near-linear plot of $\log \widehat D_k$ versus $k$ with slope $\log\rho<0$, and similar slopes across schedules; fitted slopes should vary by less than 20% and the distance should fall below $C\rho^k$ after the penalty is active. An ablation without the mixing loss should show schedule-dependent decay or occasional growth.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.