Renormalized Infinite-Depth Jacobian Regularizer

(1) Integrate this at the training-time Jacobian/stability monitor of a tied residual network, recurrent network, or state-space model. For a block update $h_{n+1}=h_n+\alpha f_{\theta}(h_n)$, obtain the Jacobian-vector product $J_nv=\partial h_{n+1}/\partial h_n\,v$ without forming a dense Jacobian. Use damping $\rho=0.90$ or $0.95$ and truncation order $M=4$--$8$. (2) Pseudocode: sample $s$ Rademacher vectors $v^{(r)}$; for each vector, propagate through Jacobian blocks using $v_{n+1}=\rho^nJ_nv_n$; estimate traces with Hutchinson, $\widehat{\tau}_m=s^{-1}\sum_r(v^{(r)})^\top T_\rho^mv^{(r)}$, using repeated Jacobian-vector products; compute $L_M=-\sum_{m=1}^{M}\widehat{\tau}_m/m$. Fit tail motif parameters from the last 20--50 block Jacobian norms or projected Jacobians, estimating dominant eigenvalues $\lambda_j$ by power iteration and coefficients $c_j$ by least squares from $\widehat{\tau}_m\approx\sum_jc_j(\rho\lambda_j)^m$. Compute $C=\sum_jc_j\operatorname{Li}_2(\rho\lambda_j)$ and add $\beta(L_M+C)^2$ to the task loss. (3) The trace expansion and dilogarithm are fixed mathematically; $\lambda_j,c_j$, $\beta$, $\rho$, and $M$ are empirical choices. Skip a counterterm when $|\rho\lambda_j|\geq0.95$ to preserve numerical convergence. (4) First test a 32-layer tied residual MLP or linear state-space model on sequential MNIST or CIFAR-10, comparing ordinary weight decay and an unregularized baseline at equal FLOPs. Record gradient explosion/vanishing, loss variance, validation accuracy, and Jacobian spectral radius. Success means fewer unstable runs and higher validation accuracy at matched training loss.