Cρ-stable recurrent transition
Implementation & benchmark of arXiv:2607.13794 — Complete functional calculus bounds for $ρ$-contractions
Source paper: Complete functional calculus bounds for $ρ$-contractions arXiv:2607.13794 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Constrain the transition matrix of an RNN or linear state-space model to the paper's class Cρ instead of controlling only its spectral radius or spectral norm. The resulting transition has an explicit dilation certificate and satisfies ∥T^n∥ ≤ ρ for every time horizon, preventing exploding hidden states while retaining nonnormal dynamics that ordinary spectral normalization may remove.
Formulas
Mathematical statement
An operator T on a Hilbert space belongs to Cρ if there is a unitary U on a larger space K and an orthogonal projection P_H onto the model state space H such that T^n = ρ P_H U^n|_H for every integer n ≥ 1. Consequently, ∥T^n∥ ≤ ρ because ∥U^n∥ = 1 and ∥P_H∥ = 1. The paper gives the intrinsic certificate I - 2(1 - 1/ρ) Re(z̅T) - (2/ρ - 1)|z|^2 T*T ≽ 0 for every complex z with |z| ≤ 1, where I is the identity, T* is the conjugate transpose, Re(B) = (B + B*)/2, and ≽ 0 means positive semidefinite. For ρ = 1 this reduces to the contraction condition ∥T∥ ≤ 1; for ρ = 2 it becomes I - Re(z̅T) ≽ 0 for every |z| ≤ 1, equivalent to numerical radius w(T) ≤ 1. Enforcing the inequality for a phase grid z = exp(iθ) gives a directly computable approximate Cρ certificate. The complete functional-calculus result additionally bounds matrix-valued analytic functions F evaluated at T, motivating stability tests for block- or channel-valued temporal filters.
Implementation notes
(1) Integration point: replace the recurrent transition h_{t+1}=Th_t+Bx_t, or the SSM state matrix discretization, with a trainable complex or real matrix T. Apply the certificate as a training-time regularizer and optionally as a projection step after each optimizer update. Use ρ=1 for a strict contraction baseline, ρ=2 for a numerical-radius-constrained model, and test intermediate values such as 1.25 and 1.5. For real T, evaluate the complex phase condition using complex arithmetic or represent T as a real 2d×2d block matrix. (2) Pseudocode: initialize raw T; at each update construct T; for j=0,...,M-1 set z=exp(i*2πj/M), Q_j=I-2*(1-1/ρ)*Re(conj(z)*T)-(2/ρ-1)*abs(z)^2*T.H*T; compute v_j=-lambda_min(Q_j); add Vρ=(1/M)Σ softplus(τv_j) to the task loss; backpropagate and update T. Every K steps, optionally subtract η times the gradient of the largest violation. (3) Compute the smallest eigenvalues with 5–10 Lanczos iterations rather than a full eigendecomposition. The phase grid approximates the all-z condition, so validate with a denser grid and directly measure max_{1≤n≤N}∥T^n∥. The theorem predicts the target observable ∥T^n∥≤ρ, but does not require constructing the dilation U. (4) First experiment: compare a vanilla linear RNN, spectral-normalized linear RNN, and Cρ-regularized linear RNN on the adding problem and permuted sequential MNIST using equal hidden width and optimizer. Log training loss, gradient norm, hidden-state norm, maximum ∥T^n∥ for n=1,...,512, and validation accuracy. Success means no exploding trajectories, lower long-horizon gradient variance, and equal-or-better accuracy at sequence lengths where the unconstrained baseline becomes unstable; ρ>1 should preserve useful memory without uncontrolled growth.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.