Unverified 2026

Residue-aware Jacobian stabilization

Implementation & benchmark of arXiv:2609.02614 — Ruelle--Pollicott Theory for Metastable Systems: A Unified Framework for Tipping Transitions

Usefulness6/10

Difficulty6/10

Novelty7/10

Dynamics Stability Generalization Rnn Ssm Training dynamics Regularization Dynamical systems Spectral theory Linear algebra Operator theory

Source paper: Ruelle--Pollicott Theory for Metastable Systems: A Unified Framework for Tipping Transitions arXiv:2609.02614 ⓘ · analyzed Sep 3, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Regularize a recurrent or continuous-time state model according to the response residues of its slow Jacobian modes, rather than penalizing every small eigenvalue equally. A mode is penalized only when it is both dynamically slow and strongly coupled to the chosen output, loss, or input perturbation, preserving useful slow memory while suppressing dangerous amplification.

Formulas

$$\mathscr{R}_{j,k}^{f}(B)=\mu\!\left(BN_j^k\Pi_j f^c\right),\qquad f^c=f-\mu(f)$$

$$\text{sensitivity contribution}_{j,k}\;\propto\;\frac{\mathscr{R}_{j,k}^{f}(B)}{(-\lambda_j)^{k+1}}$$

$$Jr_j=\lambda_jr_j,\quad l_j^\top J=\lambda_jl_j^\top,\quad l_j^\top r_j=1,\quad \Pi_j=r_jl_j^\top$$

$$\mathcal{L}_{\mathrm{res}}=\frac{1}{|\mathcal{S}|}\sum_{j\in\mathcal{S}}\left(\frac{|\widehat{R}_j|}{|\lambda_j|+\varepsilon}\right)^2,\qquad \widehat{R}_j=\frac{1}{N}\sum_{i=1}^{N}b(h_i)^\top\Pi_j\nabla_h f(h_i)$$

Mathematical statement

The paper writes the RP response residue for spectral block j and nilpotent order k as $\mathscr{R}_{j,k}^{f}(B)=\mu(BN_j^k\Pi_j f^c)$, where $\mu$ is the invariant or empirical state measure, $f^c=f-\mu(f)$ is the centered observable, $B$ is the generator perturbation, $\Pi_j$ is the spectral projector for eigenvalue $\lambda_j$, and $N_j$ is the nilpotent part of the Jordan block. The corresponding sensitivity contribution is proportional to $\mathscr{R}_{j,k}^{f}(B)(-\lambda_j)^{-(k+1)}$: small $|\lambda_j|$ permits amplification, while the residue determines whether it is actually excited. For a finite-dimensional neural ODE $\dot h=F_\theta(h,x)$, use the local Jacobian $J=\partial F_\theta/\partial h$, right and left eigenvectors $Jr_j=\lambda_jr_j$, $l_j^\top J=\lambda_jl_j^\top$, normalized by $l_j^\top r_j=1$, and projector $\Pi_j=r_jl_j^\top$. For an input perturbation vector field $b(h)$, define $(B\phi)(h)=b(h)^\top\nabla_h\phi(h)$. For scalar readout $f(h)$, estimate the empirical residue by $\widehat{R}_j=\frac{1}{N}\sum_{i=1}^{N}b(h_i)^\top\Pi_j\nabla_h f(h_i)$. The diagonalizable first-order gain is $G_j=|\widehat{R}_j|/(|\lambda_j|+\varepsilon)$; for a Jordan block of size $m_j$, include $\sum_{k=0}^{m_j-1}|\widehat{R}_{j,k}|/(|\lambda_j|+\varepsilon)^{k+1}$.

Implementation notes

Integrate this into the recurrent-state update or neural-ODE vector field, not into the classifier head. During each training interval, collect a minibatch of hidden states $h_i$ and define $f(h)$ as either the task loss contribution, a selected logit, or the scalar readout used by the downstream head. Choose $b(h)$ as the derivative of the vector field with respect to a selected input coordinate, an input-noise direction, or a random Hutchinson direction in state space. Estimate the rightmost or smallest-magnitude eigenmodes of the local Jacobian $J_i=\partial F_\theta(h_i,x_i)/\partial h_i$ using Arnoldi; for a cheap MVP use one representative minibatch Jacobian or Jacobian-vector products averaged across the batch. Compute left vectors by Arnoldi on $J^\top$, normalize each pair so $l_j^\top r_j=1$, and form $\Pi_j=r_jl_j^\top$. Estimate $\widehat R_j$ from the displayed empirical average. Add $\alpha\mathcal L_{\mathrm{res}}$ to the task loss, clipping the penalty to the slowest K modes and using stop-gradient through the eigensolver initially. Pseudocode: sample states; estimate JVP/VJP operators; run Arnoldi(J,K); pair left/right modes; compute $R_j\leftarrow N^{-1}\sum_i b_i^\top r_jl_j^\top\nabla f_i$; accumulate $(|R_j|/(|\lambda_j|+\varepsilon))^2$; backpropagate task loss plus penalty. The mathematical quantities are the projector-residue formula and inverse spectral denominator; empirical quantities are the Jacobian, eigenmodes, state measure, and perturbation directions. First test on a 1–2 layer continuous-time RNN or 64-dimensional neural ODE on sequential MNIST or permuted sequential CIFAR-10, against spectral-radius, Jacobian-norm, and no-regularizer baselines. Measure exploding gradients, sensitivity to injected input noise, long-horizon hidden-state stability, and validation accuracy at equal parameter count. Success is lower perturbation amplification and fewer unstable runs without destroying long-range-memory accuracy; a stronger result is equal accuracy with a larger stable integration step.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.