# Mean-Square Stable Neural Recurrence

- ID: 1919
- Canonical URL: https://synthcore.org/idea/1919/mean-square-stable-neural-recurrence
- API JSON: https://synthcore.org/api/idea/1919.json
- API Markdown: https://synthcore.org/api/idea/1919.md
- Verification status: mech_ok_no_baseline
- Source: [arXiv:2608.07028](https://arxiv.org/abs/2608.07028)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, optimizer, regularization, training-dynamics
- Math tags: dynamical-systems, stochastic-processes, spectral-theory, linear-algebra, control-theory
- Ratings: usefulness 8/10; difficulty 6/10; novelty 7/10

## Idea description

Treat multiplicative weight noise, quantization error, or structured parameter uncertainty in a recurrent or state-space layer as an i.i.d. random linear operator and explicitly control its second-moment growth. Add a differentiable penalty or projection based on the spectral radius of the Kronecker-lifted operator, so the network can tolerate stochastic perturbations without exploding hidden-state variance or collapsing useful memory.

## Mathematical statement

For a linear neural recurrence h_{t+1}=M_t h_t, let M_t=M_0+sum_i xi_{i,t}M_i, where xi_{i,t} are zero-mean i.i.d. random variables with covariance E[xi_t xi_t^T]=Sigma_xi, independent of h_t. Define P_t=E[h_t h_t^T]. Independence and zero mean give P_{t+1}=T(P_t), with T(P)=M_0 P M_0^T+sum_{i,j}(Sigma_xi)_{ij}M_i P M_j^T. Under vectorization, vec(P_{t+1})=K vec(P_t), where K=M_0 tensor M_0+sum_{i,j}(Sigma_xi)_{ij}M_i tensor M_j. With no additive noise, mean-square stability holds when rho(K)<1; with additive covariance Q, the fixed covariance satisfies vec(P_star)=(I-K)^{-1}vec(Q) whenever rho(K)<1. For nonlinear layers, estimate M_0 and M_i as local Jacobian or tangent operators.

## Key formulas

- $$P_{t+1}=M_0P_tM_0^\top+\sum_{i,j=1}^{p}(\Sigma_\xi)_{ij}M_iP_tM_j^\top+Q,$$
- $$\operatorname{vec}(P_{t+1})=\mathcal{K}\operatorname{vec}(P_t)+\operatorname{vec}(Q),\qquad \mathcal{K}=M_0\otimes M_0+\sum_{i,j=1}^{p}(\Sigma_\xi)_{ij}M_i\otimes M_j,$$
- $$\rho(\mathcal{K})<1\ \Longrightarrow\ \|P_t-P_\star\|\lesssim \rho(\mathcal{K})^t,\qquad \operatorname{vec}(P_\star)=(I-\mathcal{K})^{-1}\operatorname{vec}(Q).$$

## Implementation notes

Integrate this into a small GRU-free linear RNN or diagonal state-space layer first, then extend it to a nonlinear RNN using the Jacobian with respect to the hidden state. Represent stochasticity as M_t=M_0+sum_i xi_{i,t}M_i. For dropout-like channel noise, M_i is the contribution of one perturbation channel; for multiplicative parameter noise, obtain M_i by differentiating the recurrent matrix with respect to selected parameter groups. At each training interval, sample a minibatch of hidden states, compute M_0 and M_i with automatic differentiation, form K implicitly, and estimate its largest eigenvalue using 10-30 power iterations. Add lambda max(0,rho_hat(K)-(1-epsilon))^2 to the task loss. Initially stop gradients through the power-iteration vector; later use a differentiable eigensolver for small hidden dimensions. The covariance recursion and spectral condition come from the paper; local Jacobians, noise covariance, and rho are estimated empirically. Avoid explicit d^2-by-d^2 matrices using randomized Kronecker-vector products. First experiment: train a 32-unit linear RNN and tanh RNN on adding-two-numbers or sequential MNIST with Gaussian multiplicative recurrent-weight noise. Sweep noise variance and record hidden covariance for 1,000 inference steps. The predicted signature is a divergence boundary when rho_hat(K) crosses 1: below it covariance approaches a finite plateau with geometric decay rate approximately rho_hat(K), while above it trace covariance grows exponentially. The measured transition should agree within 20%.

## Verification

- Status: mech_ok_no_baseline
- Mechanism evidence: yes
- Mechanism confirmed: yes
- Practical verdict: inconclusive
- Verdict: Built an exact Kronecker-lift covariance verifier and a small noisy scalar-RNN training comparison in experiment.py. The predicted stability boundary was sigma*=0.600; the sweep transitioned between sigma=0.59 (decay) and 0.61 (growth), agreeing within 3.3%. Additive-noise plateaus matched q/(1-rho) with at most 0.25% error, and covariance scaling was exact to 4.4e-16. The penalty reduced learned rho from 0.9778 to 0.9719 and the long-run second-moment proxy by 45.5%, at a small task-loss increase, so the mechanism manifested although this is not a broad accuracy win.

### Mechanism check

- Verdict: Built an exact Kronecker-lift covariance verifier and a small noisy scalar-RNN training comparison in experiment.py. The predicted stability boundary was sigma*=0.600; the sweep transitioned between sigma=0.59 (decay) and 0.61 (growth), agreeing within 3.3%. Additive-noise plateaus matched q/(1-rho) with at most 0.25% error, and covariance scaling was exact to 4.4e-16. The penalty reduced learned rho from 0.9778 to 0.9719 and the long-run second-moment proxy by 45.5%, at a small task-loss increase, so the mechanism manifested although this is not a broad accuracy win.
- Confidence: 9/10
- Limitations: Only a scalar recurrence and tiny synthetic sequence-sum task were tested; no 32-unit nonlinear RNN, sequential MNIST, structured parameter groups, implicit Kronecker power iteration, or large-scale accuracy/generalization study was performed. Monte Carlo estimates near the instability boundary are heavy-tailed and noisier than the exact recursion.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.69925
- Paired wins: 4/8
- Benchmark verdict: no significant win

## Artifacts

- [bench_report.json](https://synthcore.org/code/708/bench_report.json)
- [experiment.py](https://synthcore.org/code/708/experiment.py)
- [report.md](https://synthcore.org/code/708/report.md)
- [report_bench_2026-09-02T230453.md](https://synthcore.org/code/708/report_bench_2026-09-02T230453.md)
- [results.json](https://synthcore.org/code/708/results.json)
- [stage2_bench.py](https://synthcore.org/code/708/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/708)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.