# Density-Stable Recurrent Dynamics

- ID: 2054
- Canonical URL: https://synthcore.org/idea/2054/density-stable-recurrent-dynamics
- API JSON: https://synthcore.org/api/idea/2054.json
- API Markdown: https://synthcore.org/api/idea/2054.md
- Verification status: unverified
- Source: [arXiv:2608.09694](https://arxiv.org/abs/2608.09694)
- Category: dynamics
- Solves: stability, accuracy, generalization
- ML areas: rnn, ssm, training-dynamics
- Math tags: functional-analysis, dynamical-systems, probability
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace a strict spectral-radius or per-step activation constraint in a linear recurrent/state-space transition with a density-of-spikes constraint. Penalize the fraction of rollout times whose hidden-state norm exceeds a threshold, making the model tolerant of occasional useful transients while suppressing persistent or frequent amplification.

## Mathematical statement

For a bounded linear operator T on a Banach space X, the paper defines the uniform high-orbit-density function \(\Phi_T(R)=\sup_{\|x\|\le 1}\sup_{N\ge 1}\frac{1}{N}\operatorname{card}\{1\le j\le N:\|T^j x\|>R\}\). Mean-L-stability is equivalent to uniform boundedness in density, namely \(\Phi_T(R)\to 0\) as \(R\to\infty\). The paper further gives \(\|T^n\|=O(n)\) on arbitrary Banach spaces and, on Hilbert spaces, \(\|T^n\|=O(n^{1/2-c_T})\) for some operator-dependent constant \(c_T>0\). Here T is the recurrent transition matrix, \(T^j x\) is the hidden state after j homogeneous transitions from initial state x, R is a normalized state-norm threshold, N is rollout length, and \(\Phi_T(R)\) is approximated over sampled unit or normalized states rather than computed exactly. The transferable property is that rare exceptional times are allowed, but their asymptotic density must vanish; this is less restrictive than demanding \(\|T^j\|\le 1\) for every j and more informative than checking only eigenvalues when T is non-normal.

## Key formulas

- $$\Phi_{T}(R)=\sup_{\|x\|\leq 1}\sup_{N\geq 1}\frac{1}{N}\operatorname{card}\{1\leq j\leq N:\|T^{j}x\|>R\}.$$
- $$T\text{ is mean-L-stable}\quad\Longleftrightarrow\quad\Phi_{T}(R)\longrightarrow 0\quad\text{as }R\to\infty.$$
- $$\|T^{n}\|=O(n)\quad\text{on Banach spaces},\qquad \|T^{n}\|=O\!\left(n^{1/2-c_T}\right)\quad\text{on Hilbert spaces for some }c_T>0.$$
- $$\widehat\Phi_T(R)=\max_{k=1}^{K}\max_{N\in\mathcal N}\frac{1}{N}\sum_{j=1}^{N}\mathbf 1\!\left[\frac{\|T^j x_k\|_2}{\|x_k\|_2+\varepsilon}>R\right],\qquad \mathcal L_{\mathrm{density}}=\sum_{R\in\mathcal R}w_R\,\max(0,\widehat\Phi_T(R)-\delta_R)^2.$$

## Implementation notes

(1) Integration point: use this on the transition matrix T of a linear RNN or state-space model, such as h_{t+1}=T h_t+B u_t, not on the nonlinear input projection. Apply the penalty during training every M optimizer steps, and optionally use it to trigger a mild rescaling of T. Normalize hidden states by the initial probe norm so the threshold is independent of probe magnitude.

(2) Pseudocode:
```text
sample K probe vectors x[k] ~ N(0,I), normalize x[k] /= ||x[k]||2
for each x[k]:
    z = x[k]
    for j = 1,...,Nmax:
        z = T @ z
        record q[k,j] = ||z||2 / (||x[k]||2 + eps)
for R in thresholds:
    phi_hat[R] = max_k,max_N mean(q[k,1:N] > R)
L_density = sum_R w[R] * relu(phi_hat[R] - delta[R])**2
loss = task_loss + lambda_density * L_density
backpropagate(loss)
if max(q) > hard_cap: T <- T * (hard_cap / max(q))**alpha
```
The indicator can be replaced during backpropagation by a sigmoid \(\sigma((q-R)/\tau)\), while the exact indicator is retained for evaluation. The formula for \(\widehat\Phi_T\) is the sampled implementation of the paper's \(\Phi_T\); K random probes, several rollout lengths \(N\), thresholds \(R\), and tolerances \(\delta_R\) are engineering approximations to the suprema and limit.

(3) Compute directly from the current T powers, preferably with matrix-vector products rather than forming T^j. Estimate worst directions by adding power-iteration probes initialized from random vectors and retaining the probe with the largest density. The paper's rigorous statements provide the qualitative safety target and the O(n) or Hilbert-space sublinear power-growth reference; they do not provide a universal numerical value for \(c_T\), so do not impose one. Log spectral radius, largest singular value, \(\widehat\Phi_T(R)\), maximum transient norm, and task loss separately.

(4) First experiment: train a 2-layer linear-plus-gated RNN or diagonal/state-space sequence model on sequential MNIST and the copy/addition task. Compare unconstrained training, spectral normalization of T, and the density penalty at matched parameter count and FLOPs. Use 256-step and 2048-step rollouts, including deliberately non-normal upper-triangular initializations. Success means fewer NaNs and exploding trajectories, lower long-horizon gradient variance, and equal or better validation accuracy than strict spectral normalization; additionally test whether the density method preserves occasional transient amplification that improves copy-task accuracy. A useful falsification criterion is no improvement in long-rollout stability or a substantial increase in training cost without accuracy benefit.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.