Density-Stable Recurrent Dynamics

(1) Integration point: use this on the transition matrix T of a linear RNN or state-space model, such as h_{t+1}=T h_t+B u_t, not on the nonlinear input projection. Apply the penalty during training every M optimizer steps, and optionally use it to trigger a mild rescaling of T. Normalize hidden states by the initial probe norm so the threshold is independent of probe magnitude.

(2) Pseudocode:

sample K probe vectors x[k] ~ N(0,I), normalize x[k] /= ||x[k]||2
for each x[k]:
    z = x[k]
    for j = 1,...,Nmax:
        z = T @ z
        record q[k,j] = ||z||2 / (||x[k]||2 + eps)
for R in thresholds:
    phi_hat[R] = max_k,max_N mean(q[k,1:N] > R)
L_density = sum_R w[R] * relu(phi_hat[R] - delta[R])**2
loss = task_loss + lambda_density * L_density
backpropagate(loss)
if max(q) > hard_cap: T <- T * (hard_cap / max(q))**alpha

The indicator can be replaced during backpropagation by a sigmoid \(\sigma((q-R)/\tau)\), while the exact indicator is retained for evaluation. The formula for \(\widehat\Phi_T\) is the sampled implementation of the paper's \(\Phi_T\); K random probes, several rollout lengths \(N\), thresholds \(R\), and tolerances \(\delta_R\) are engineering approximations to the suprema and limit.

(3) Compute directly from the current T powers, preferably with matrix-vector products rather than forming T^j. Estimate worst directions by adding power-iteration probes initialized from random vectors and retaining the probe with the largest density. The paper's rigorous statements provide the qualitative safety target and the O(n) or Hilbert-space sublinear power-growth reference; they do not provide a universal numerical value for \(c_T\), so do not impose one. Log spectral radius, largest singular value, \(\widehat\Phi_T(R)\), maximum transient norm, and task loss separately.

(4) First experiment: train a 2-layer linear-plus-gated RNN or diagonal/state-space sequence model on sequential MNIST and the copy/addition task. Compare unconstrained training, spectral normalization of T, and the density penalty at matched parameter count and FLOPs. Use 256-step and 2048-step rollouts, including deliberately non-normal upper-triangular initializations. Success means fewer NaNs and exploding trajectories, lower long-horizon gradient variance, and equal or better validation accuracy than strict spectral normalization; additionally test whether the density method preserves occasional transient amplification that improves copy-task accuracy. A useful falsification criterion is no improvement in long-rollout stability or a substantial increase in training cost without accuracy benefit.