# Renormalized Infinite-Depth Jacobian Regularizer

- ID: 181
- Canonical URL: https://synthcore.org/idea/181/renormalized-infinite-depth-jacobian-regularizer
- API JSON: https://synthcore.org/api/idea/181.json
- API Markdown: https://synthcore.org/api/idea/181.md
- Verification status: unverified
- Source: [arXiv:2608.25786](https://arxiv.org/abs/2608.25786)
- Category: regularization
- Solves: stability, generalization
- ML areas: ssm, rnn, training-dynamics, regularization
- Math tags: dynamical-systems, spectral-theory, functional-analysis
- Ratings: usefulness 5/10; difficulty 7/10; novelty 8/10

## Idea description

Treat repeated residual blocks as an infinite directed transition system, damp transitions according to their depth, and regularize a finite part of the resulting Fredholm log-determinant. Subtracting a dilogarithmic counterterm prevents the regularizer from being dominated by infinitely repeated short cycles, while retaining information about global recurrent amplification.

## Mathematical statement

The paper uses the classical dilogarithm $\operatorname{Li}_{2}(z)=\sum_{k\geq1}z^{k}/k^{2}=-\int_{0}^{z}\log(1-t)/t\,dt$ and states that a height-damped transition operator is trace class, so its determinant can be treated through a normally convergent logarithmic expansion. For a trace-class operator $T$ with spectral radius below $1$, the Fredholm identity is $\log\det(I-uT)=-\sum_{m\geq1}u^{m}\operatorname{tr}(T^{m})/m$. In an infinite repeated architecture, let $J_n$ be the Jacobian transition at height/layer $n$, let $\rho\in(0,1)$ be a height damping factor, and define the block transition operator $T_\rho$ by $(T_\rho x)_{n+1}=\rho^{n}J_nx_n$. The damping makes long-height contributions summable. If a motif has asymptotic eigenvalue $\lambda$ and multiplicity coefficient $c$, its integrated repeated-cycle contribution is represented by $c\operatorname{Li}_2(\rho\lambda)$; subtracting these terms is the neural finite-part analogue of the paper's integrated-pressure counterterm. The practical regularizer is $R=\left|\log\det(I-T_\rho)+\sum_jc_j\operatorname{Li}_2(\rho\lambda_j)\right|^2$, with $\lambda_j,c_j$ estimated from tail Jacobian statistics.

## Key formulas

- $$\operatorname{Li}_{2}(z)=\sum_{k\geq 1}\frac{z^{k}}{k^{2}}=-\int_{0}^{z}\frac{\log(1-t)}{t}\,dt,\qquad |z|<1.$$
- $$\log\det(I-uT)=-\sum_{m\geq1}\frac{u^{m}}{m}\operatorname{tr}(T^{m}),\qquad \rho(uT)<1.$$
- $$(T_{\rho}x)_{n+1}=\rho^{n}J_nx_n,\qquad 0<\rho<1.$$
- $$R_{\mathrm{ren}}=\left|-\sum_{m=1}^{M}\frac{\widehat{\tau}_m}{m}+\sum_{j=1}^{K}c_j\operatorname{Li}_{2}(\rho\lambda_j)\right|^{2},\qquad \widehat{\tau}_m\approx\operatorname{tr}(T_{\rho}^{m}).$$

## Implementation notes

(1) Integrate this at the training-time Jacobian/stability monitor of a tied residual network, recurrent network, or state-space model. For a block update $h_{n+1}=h_n+\alpha f_{\theta}(h_n)$, obtain the Jacobian-vector product $J_nv=\partial h_{n+1}/\partial h_n\,v$ without forming a dense Jacobian. Use damping $\rho=0.90$ or $0.95$ and truncation order $M=4$--$8$. (2) Pseudocode: sample $s$ Rademacher vectors $v^{(r)}$; for each vector, propagate through Jacobian blocks using $v_{n+1}=\rho^nJ_nv_n$; estimate traces with Hutchinson, $\widehat{\tau}_m=s^{-1}\sum_r(v^{(r)})^\top T_\rho^mv^{(r)}$, using repeated Jacobian-vector products; compute $L_M=-\sum_{m=1}^{M}\widehat{\tau}_m/m$. Fit tail motif parameters from the last 20--50 block Jacobian norms or projected Jacobians, estimating dominant eigenvalues $\lambda_j$ by power iteration and coefficients $c_j$ by least squares from $\widehat{\tau}_m\approx\sum_jc_j(\rho\lambda_j)^m$. Compute $C=\sum_jc_j\operatorname{Li}_2(\rho\lambda_j)$ and add $\beta(L_M+C)^2$ to the task loss. (3) The trace expansion and dilogarithm are fixed mathematically; $\lambda_j,c_j$, $\beta$, $\rho$, and $M$ are empirical choices. Skip a counterterm when $|\rho\lambda_j|\geq0.95$ to preserve numerical convergence. (4) First test a 32-layer tied residual MLP or linear state-space model on sequential MNIST or CIFAR-10, comparing ordinary weight decay and an unregularized baseline at equal FLOPs. Record gradient explosion/vanishing, loss variance, validation accuracy, and Jacobian spectral radius. Success means fewer unstable runs and higher validation accuracy at matched training loss.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.