{
 "artifacts": null,
 "category": "regularization",
 "description": "Regularize an RNN or linear state-space layer using the spectrum of its stationary hidden-state covariance, not merely the eigenvalues or spectral radius of its recurrent matrix. The goal is to prevent a stable but non-normal transition matrix from amplifying noise and inputs into a few dominant principal components, while preserving recurrent memory.",
 "formulas_latex": [
  "$$\\mathbf{x}_{t+1}=\\mathbf{J}\\mathbf{x}_{t}+\\mathbf{z}_{t},\\qquad \\mathbf{z}_{t}\\sim\\mathcal{N}(0,\\mathbf{I}_N).$$",
  "$$\\mathbf{Q}=\\mathbf{J}\\mathbf{Q}\\mathbf{J}^{\\top}+\\mathbf{I}_N=\\sum_{k=0}^{\\infty}\\mathbf{J}^{k}(\\mathbf{J}^{\\top})^{k},\\qquad \\rho(\\mathbf{J})\u003c1.$$",
  "$$p_i=\\frac{\\lambda_i(\\mathbf{Q})}{\\sum_{j=1}^{N}\\lambda_j(\\mathbf{Q})},\\qquad H(\\mathbf{Q})=-\\sum_{i=1}^{N}p_i\\log(p_i+\\epsilon),\\qquad r_{\\mathrm{eff}}(\\mathbf{Q})=e^{H(\\mathbf{Q})}.$$",
  "$$\\mathcal{L}_{\\mathrm{total}}=\\mathcal{L}_{\\mathrm{task}}+\\beta\\left[\\log r_{\\mathrm{target}}-H(\\widehat{\\mathbf{Q}})\\right]_+^2+\\gamma\\left[\\rho(\\mathbf{J})-\\rho_{\\mathrm{max}}\\right]_+^2.$$"
 ],
 "id": 289,
 "implementation": "Integrate the method into the hidden-state update of a small tanh RNN, linear RNN, or diagonal-plus-low-rank SSM. The exact integration point is the recurrent state covariance, computed from the batch of hidden states after the recurrent update and before the output projection. Maintain an exponential moving average $\\widehat{Q}_t=\\alpha\\widehat{Q}_{t-1}+(1-\\alpha)C_t$, where $C_t=(1/B)\\sum_{b=1}^{B}(h_{b,t}-\\bar h_t)(h_{b,t}-\\bar h_t)^\\top+\\delta I$; $B$ is batch size, $h_{b,t}$ is the hidden state, $\\bar h_t$ is the batch mean, and $\\delta$ prevents singularity. Every $K$ optimizer steps, symmetrize $\\widehat Q$, compute its eigenvalues with `torch.linalg.eigvalsh`, normalize them into $p_i$, and add $\\beta[\\log r_{target}-H(\\widehat Q)]_+^2$ to the task loss. Detach the EMA update if needed, but retain gradients through the current covariance estimate for the regularizer. The Lyapunov series is the mathematical interpretation; initially estimate $Q$ from activations rather than explicitly solving the equation. For large hidden dimensions, use randomized Lanczos estimates of the leading eigenvalues and stochastic trace estimates. Estimate $\\rho(J)$ with 5--10 power iterations and optionally add the spectral-radius barrier only when it activates. First test on permuted-MNIST and the adding problem with a 128-unit tanh RNN. Compare plain training, spectral-radius regularization, and covariance regularization at matched FLOPs. Record accuracy, hidden-state effective rank, top-eigenvalue share, gradient norms, and failure rate from exploding or vanishing gradients. Success means reduced variance concentration and fewer unstable runs at equal accuracy, or better long-sequence accuracy at equal parameter count. Sweep $\\beta$, $\\alpha\\in\\{0.95,0.99\\}$, and target effective-rank fractions of 0.25, 0.5, and 0.75 of the hidden dimension.",
 "math_summary": "For the discrete dynamics, $\\mathbf{x}_{t+1}=\\mathbf{J}\\mathbf{x}_t+\\mathbf{z}_t$, with $\\mathbf{z}_t\\sim\\mathcal{N}(0,\\mathbf{I}_N)$, stationarity requires the spectral radius $\\rho(\\mathbf{J})\u003c1$. The stationary covariance $\\mathbf{Q}=\\mathbb{E}[\\mathbf{x}_t\\mathbf{x}_t^\\top]$ obeys the discrete Lyapunov equation $\\mathbf{Q}=\\mathbf{J}\\mathbf{Q}\\mathbf{J}^\\top+\\mathbf{I}_N$ and equivalently $\\mathbf{Q}=\\sum_{k=0}^{\\infty}\\mathbf{J}^k(\\mathbf{J}^\\top)^k$. Here $\\mathbf{J}\\in\\mathbb{R}^{N\\times N}$ is the recurrent matrix, $\\mathbf{I}_N$ is the identity noise covariance, and $\\mathbf{Q}$ is positive definite. Let $\\lambda_1,\\ldots,\\lambda_N$ be the eigenvalues of $\\mathbf{Q}$ and $p_i=\\lambda_i/(\\sum_j\\lambda_j)$ their normalized variance shares. The spectral entropy $H(Q)=-\\sum_i p_i\\log(p_i+\\epsilon)$ measures how evenly variance is distributed, and the effective rank is $r_{\\mathrm{eff}}(Q)=\\exp(H(Q))$. Penalizing low entropy discourages covariance collapse into a few amplified directions, while the Lyapunov series explains why non-normal products $\\mathbf{J}^k(\\mathbf{J}^\\top)^k$ can create collapse even when $\\rho(\\mathbf{J})\u003c1$.",
 "math_tags": [
  "random-matrix",
  "spectral-theory",
  "linear-algebra",
  "dynamical-systems"
 ],
 "ml_areas": [
  "rnn",
  "ssm",
  "regularization",
  "training-dynamics"
 ],
 "paper": {
  "arxiv_id": "2606.31944",
  "arxiv_url": "https://arxiv.org/abs/2606.31944",
  "summary_what_math_gives_to_ml": "The paper isolates a failure mode of random recurrent dynamics that is invisible from eigenvalues alone: a stable, strongly non-normal recurrent matrix can produce a highly anisotropic stationary covariance and concentrate activity into a few principal components. The transferable object is the stationary covariance equation, whose spectrum directly measures how recurrent amplification distributes noise and inputs across hidden directions. A practical neural-network adaptation is to monitor or estimate this covariance spectrum in an RNN and penalize excessive spectral concentration, rather than controlling only the spectral radius of the recurrent matrix. This should improve stability and usable hidden-state dimensionality without requiring the recurrent matrix itself to be normal or orthogonal.",
  "title": "Stationary covariance spectra of discrete-time non-normal random recurrent dynamics",
  "year": "2026"
 },
 "ratings": {
  "difficulty": 4,
  "novelty": 5,
  "usefulness": 6
 },
 "solves": [
  "stability",
  "generalization",
  "accuracy"
 ],
 "title": "Non-normal covariance equalizer",
 "url": "https://synthcore.org/idea/289/non-normal-covariance-equalizer",
 "verification": {
  "peer_reviewed": false,
  "status": "unverified",
  "status_label": "Unverified",
  "verdict_source": "deterministic test code (paired-seed permutation statistics)",
  "verification_axes": {
   "benchmark_mechanism": {
    "confirmed": null,
    "tested": false
   },
   "practical_benchmark": {
    "beats_baseline": null,
    "tested": false
   },
   "toy_mechanism_gate": {
    "confirmed": null,
    "tested": false
   }
  }
 }
}