✓✓ Beats tuned baseline 2026

Mean-square-stable Markov-switching recurrent layer

Implementation & benchmark of arXiv:2607.09994 — Spectral Characterisation of Covariance Existence in Markov-Switching Affine Recurrences

Usefulness8/10

Difficulty6/10

Novelty7/10

Dynamics Stability Accuracy Generalization Rnn Ssm Moe routing Training dynamics Dynamical systems Spectral theory Linear algebra Probability

Source paper: Spectral Characterisation of Covariance Existence in Markov-Switching Affine Recurrences arXiv:2607.09994 ⓘ · analyzed Aug 30, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace an unconstrained recurrent or state-space transition with a finite set of mode matrices selected by a Markov routing process, while explicitly constraining the associated Kronecker operator to have spectral radius below one. This targets exploding hidden-state variances caused by rare but repeatedly visited unstable modes, a failure mode not detected by average spectral radius or ordinary Lyapunov stability.

Formulas

$$Z_{n+1}=A_{\Theta_{n+1}}Z_n+\zeta_{n+1}$$

$$T_{ji}:=p_{ij}(A_j\otimes A_j)$$

$$\rho(T)<1$$

$$\widetilde A_j=\frac{A_j}{\sqrt{\max(\rho(T),\,\varepsilon)}}$$

Mathematical statement

The paper studies the affine recurrence $Z_{n+1}=A_{\Theta_{n+1}}Z_n+\zeta_{n+1}$, where $\Theta_n\in\{1,\ldots,K\}$ is a Markov chain with transition probabilities $p_{ij}=\Pr(\Theta_{n+1}=j\mid\Theta_n=i)$, $A_j\in\mathbb{R}^{d\times d}$ is the transition matrix for mode $j$, and $\zeta_n$ is innovation noise. It defines the block Markov-switching Kronecker operator $T$ through blocks $T_{ji}:=p_{ij}(A_j\otimes A_j)$, where $A_j\otimes A_j\in\mathbb{R}^{d^2\times d^2}$ and $\rho(T)$ is the spectral radius of the resulting $Kd^2\times Kd^2$ matrix. Under a Perron-excitation condition on the innovation covariance, $\rho(T)<1$ is necessary and sufficient for a stationary solution to have finite second moments. We transplant this as a trainable mean-square stability constraint: the mode-dependent hidden transition must satisfy $\rho(T)\le 1-\varepsilon$, with $\varepsilon>0$ a safety margin. Unlike merely enforcing $\max_j\rho(A_j)<1$, this condition incorporates mode persistence $p_{ij}$ and the squared amplification induced by $A_j\otimes A_j$.

Implementation notes

Integrate this into a gated RNN or linear state-space layer whose hidden state is $h_t\in\mathbb{R}^d$ and whose router chooses one of $K$ transition matrices $A_j$. Use the actual mode probabilities $p_{ij}$ if the router is Markovian; for a simpler first test, estimate them from consecutive hard router decisions with Laplace smoothing. The forward recurrence is $h_{t+1}=A_{s_{t+1}}h_t+B_{s_{t+1}}x_{t+1}+b_{s_{t+1}}$. Build a block operator with block $(j,i)$ equal to $p_{ij}(A_j\otimes A_j)$, exactly matching the paper's $T_{ji}$. Do not materialize this matrix for large $d$: estimate its largest eigenvalue with 10--20 power iterations using the implicit map $V_i\mapsto\sum_j p_{ij}(A_j\otimes A_j)V_i$, implemented through the identity $(A\otimes A)\operatorname{vec}(X)=\operatorname{vec}(AXA^\top)$, where each $V_i$ is represented as a $d\times d$ matrix. Add $\lambda\max(0,\widehat\rho(T)-(1-\varepsilon))^2$ to the training loss, or rescale all $A_j$ by $1/\sqrt{\max(\widehat\rho(T),\varepsilon)}$ after each optimizer step. The exact mathematical quantity is $\rho(T)$; the empirical quantities are the transition matrix $p$ estimated from router traces and the finite-iteration eigenvalue estimate. First test on a copy task and sequential MNIST with a 2-layer GRU-like model or diagonal SSM, comparing unconstrained mode matrices, per-mode spectral normalization, and this Markov-Kronecker constraint at equal parameter count. Log hidden-state covariance, gradient norm, validation accuracy, and the estimated $\rho(T)$; success means fewer exploding-gradient runs, bounded long-context hidden variance, and better accuracy at sequence lengths where the unconstrained baseline becomes unstable.

Verification

Beats tuned baseline

Stage 1 · Toy mechanism gate: Passed ✓

Stage 2 · Mechanism transferred to benchmark: Confirmed ✓

Stage 2 · Practical benchmark result: Significant win ✓

Methodology: 8 paired seeds, permutation test (p<0.05) against a learning-rate-tuned baseline. How verification works

Stage 1 — Mechanism check agent confidence 9/10

Built a reproducible toy implementation of the Markov-switching Kronecker operator, implicit power iteration, exact second-moment recursion, and Monte Carlo recurrence comparison. The mechanism manifested clearly: rho(T) scaled exactly as gamma^2, the second-moment boundary occurred at rho(T)=1, and increasing mode persistence increased rho(T) from 0.875 to 1.547. Rescaling the unstable baseline from rho=1.769 to rho=0.941 reduced final Monte Carlo state MSE from 3.12e39 to 9.26, although no neural-network task was tested.

Agent confidence: 9/10

Baseline: Rare unstable mode with persistence 0.97: rho(T)=1.7686; final Monte Carlo E[h^2]=3.12e39.
Idea: Kronecker rescaling: rho(T)=0.9409; final Monte Carlo E[h^2]=9.26. Gain-scaling relative error was <=1.23e-16; rho(T)>1 cases showed explosive exact moments while rho(T)<1 cases remained bounded; persistence sweep rho(T): 0.875, 1.026, 1.284, 1.486, 1.547.

Limitations: This is a scalar two-mode toy recurrence rather than a trainable gated RNN or SSM, and it does not test copy-task accuracy, sequential MNIST, optimization dynamics, learned routing, GPU behavior, or parameter/FLOP-matched neural baselines. The implementation uses exact dense operators only for tiny scalar modes; larger-matrix implicit Kronecker code was not benchmarked.

How to run: python3 run_experiment.py

Stage 2 — Benchmark vs. tuned baseline (arXiv:2607.09994) Sep 2, 2026

Beats the tuned baseline by 21.1% (p=0.0163, wins 7 / 8 paired seeds; metric: lower is better).

Benchmark: Dynamics forecasting
Model: rnn_small
Paired seeds: 8
Baseline mean: 0.0126
Idea mean: 0.0099
Effect (Δ): -0.0027 (−21.1%; negative = idea better)
Wins: 7 / 8 paired seeds
p-value: 0.0163 (permutation test, 20 000 shuffles)
Smallest detectable effect: ±9.1%
Mechanism: Confirmed ✓
Practical effect: Helps
Baseline tuning: swept over 3 configs

Limitations:

The requested standard dynamics dataset size and epoch budget were reduced to 800 training samples, 200 test samples, and 6 epochs after the initial 4000/1000, 20-epoch run exceeded the runtime limit; this reduced configuration was applied identically to both systems. The model is a small diagonal Markov-switching recurrent layer rather than the built-in GRU implementation, although baseline and idea share the same custom base architecture and differ only in the stability projection. No CIFAR, sequence, or tabular tracks were tested, and long-horizon rollout error beyond the built-in one-step target was not evaluated.

How to run: OMP_NUM_THREADS=4 MKL_NUM_THREADS=4 /home/maxwelhelp/main/bin/python3 bench_run.py

Verdict computed by deterministic test code from paired-seed statistics — not by the language model.

Stage 2 — Benchmark Aug 30, 2026 latest Worked ✓

Agent confidence: 8/10

Built a Markov-switching diagonal recurrent layer for the structurally matched dynamics track, with an implicit/exact diagonal Kronecker spectral-radius calculation and post-step projection to rho(T)<=0.90. Across 8 paired seeds, the idea achieved lower test MSE than the swept baseline (0.00995 vs 0.01261), with delta=-0.00266 and permutation p=0.0163, so the bench verdict is idea better (significant). The trained-model mechanism signature was confirmed: mean effective rho(T)=0.8995 and finite hidden second moment=0.4346.

Baseline: Dynamics/rnn_small matched baseline, lr=0.006 selected by sweep: mean test MSE 0.0126062, std 0.0026332, per-seed [0.0118508, 0.0137482, 0.0138083, 0.0132037, 0.0133342, 0.0063090, 0.0125824, 0.0160131].
Idea: Markov-Kronecker projected recurrent layer, lr=0.006 selected from the parity-matched idea sweep: mean test MSE 0.0099479, std 0.0027885, per-seed [0.0078573, 0.0103606, 0.0140214, 0.0103935, 0.0113713, 0.0041571, 0.0092710, 0.0121511].
Idea wins: 7
N pairs: 8
Paired delta mean: -0.0026582953287288547
Permutation p value: 0.0163

Benchmark result