Empirical-Covariance-Weighted Low-Rank Dynamics

Integrate this into the recurrent transition or SSM state-update matrix, not into the input/output projections. For a layer with $h_{t+1}=f(Ah_t+Bu_t)$, maintain an exponential-moving covariance estimate instead of storing all activations: initialize $C=\epsilon I$ and update once per minibatch using $C\leftarrow\rho C+(1-\rho)H^\top H/B$, where $H\in\mathbb{R}^{B\times d}$ contains pre-transition hidden states and $B$ is the number of collected states. Form $C_\lambda=C+\lambda I$. Every $K$ optimizer steps, compute a Cholesky factor $L$ satisfying $LL^\top=C_\lambda$ and define $M=AL$. Perform the ordinary task-gradient update on $A$, then apply a proximal penalty in the whitened variable: compute an SVD $M=U\operatorname{diag}(s)V^\top$, replace each singular value by $s_i'=\max(s_i-\eta\alpha,0)/(1+\eta\beta)$, set $M'=U\operatorname{diag}(s')V^\top$, and map back with $A\leftarrow M'L^{-1}$. If exact proximal mapping is too expensive, use the differentiable surrogate gradient of $\|AL\|_*$ from the SVD and apply it jointly with AdamW. The mathematics determines the covariance weighting, singular-value shrinkage, and ridge denominator; $\rho$, $\lambda$, $\alpha$, $\beta$, and update frequency are empirical hyperparameters. First test on Copying Memory and permuted sequential MNIST using a two-to-four-layer linear or nonlinear SSM. Compare unconstrained AdamW, ordinary nuclear regularization $\|A\|_*$, and the proposed $\|A\widehat C_\lambda^{1/2}\|_*$. Measure validation loss, hidden-state explosion or vanishing, effective rank, rollout stability, and transition-matrix memory. A useful success signal is equal or lower loss with fewer nonzero singular values, fewer unstable long-sequence rollouts, and improved loss at fixed transition-matrix rank.