# Perron-Critical Sparse Routing

- ID: 2644
- Canonical URL: https://synthcore.org/idea/2644/perron-critical-sparse-routing
- API JSON: https://synthcore.org/api/idea/2644.json
- API Markdown: https://synthcore.org/api/idea/2644.md
- Verification status: unverified
- Source: [arXiv:2608.21125](https://arxiv.org/abs/2608.21125)
- Category: architecture
- Solves: stability, scalability, speedup
- ML areas: moe-routing, transformer, training-dynamics, optimizer
- Math tags: spectral-theory, stochastic-processes, probability, functional-analysis
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Model dynamic routing as a multitype branching process: an active token of type d probabilistically creates child activations of type d'. Estimate the corresponding mean offspring operator and regulate its Perron root to a target reproduction rate, typically near one. This should make adaptive-depth or recursively routed networks use sparse computation without producing either rapidly vanishing paths or uncontrolled activation explosions.

## Mathematical statement

The paper defines the tilted mean matrix M(\lambda) by M(\lambda)_{d,d'}=e^{-\lambda}M_{d,d'} and the free energy \Lambda(\lambda)=\log\rho(M(\lambda)), where d and d' are parent and child types, M_{d,d'} is the expected number of type-d' children from a type-d parent, \lambda is a spatial/depth tilt, and \rho denotes the Perron spectral radius. In the continuous-type setting, the analogous mean offspring operator is K on functions of the type variable A, with (Kf)(a)=\int k(a,a')f(a')\,\mu(da'); here a and a' are parent and child strengths, k is the expected child-intensity kernel, and \mu is the reference strength measure. The paper's quasi-compactness and spectral-gap mechanism means that K has a dominant positive eigenvalue \rho(K) separated from the rest of the spectrum; if r(a)>0 and \ell(a)>0 are right and left Perron eigenfunctions normalized by \int \ell(a)r(a)\mu(da)=1, then Kr=\rho(K)r and K^*\ell=\rho(K)\ell. Use \rho(K) as the neural-network route reproduction factor. A practical loss is L_{crit}=(\log \widehat{\rho}-\log\rho_{target})^2, with \rho_{target}\approx1 for stable adaptive depth or a measured budget-dependent value below one for finite computation. The scalar tilt gives \Lambda(\lambda)=\log\rho(M)-\lambda, so a desired exponential decay or growth rate can equivalently be imposed through \lambda=\log\rho(M)-\Lambda_{target}.

## Key formulas

- $$M(\lambda)_{d,d^{\prime}}:=e^{-\lambda}M_{d,d^{\prime}},\qquad \Lambda(\lambda):=\log\rho\bigl(M(\lambda)\bigr)=\log\rho(M)-\lambda.$$
- $$(Kf)(a)=\int k(a,a')f(a')\,\mu(da'),\qquad Kr=\rho(K)r,\qquad K^{*}\ell=\rho(K)\ell,\qquad \int \ell(a)r(a)\,\mu(da)=1.$$
- $$\mathcal{L}_{\mathrm{crit}}=\left(\log\widehat{\rho}-\log\rho_{\mathrm{target}}\right)^2,\qquad \widehat{\rho}\approx\frac{\ell^{\top}\widehat{M}r}{\ell^{\top}r}.$$
- $$p_{d\to d'}^{\mathrm{new}}=\operatorname{clip}\!\left(p_{d\to d'}\exp\left[-\eta_{r}\,\frac{\partial \mathcal{L}_{\mathrm{crit}}}{\partial p_{d\to d'}}\right],\epsilon,1-\epsilon\right).$$

## Implementation notes

Integrate this into a top-k MoE router, adaptive-depth Transformer, or recurrent module at the router/gating operation, not into the ordinary token loss. Assign every parent token a type d: initially use a discretized confidence, entropy, norm, or predicted difficulty score; a continuous-strength version can use binned scalar A. A routed token creates one child activation for every selected expert, recurrent block, or next-depth state, so the empirical offspring matrix is \widehat M_{ij}=    ext{number of active children in bin }j/    ext{number of active parents in bin }i. Maintain exponentially smoothed counts over each minibatch. Every T steps, compute the dominant right Perron vector r by 5-10 power iterations, r\leftarrow\widehat M r/\|\widehat M r\|_1, and estimate \widehat\rho=(\ell^T\widehat M r)/(\ell^Tr); \ell can be obtained by power iteration on \widehat M^T, or omitted in the first MVP using the Rayleigh quotient \|\widehat M r\|_1/\|r\|_1. Add \mathcal L_{crit} to the training loss, or update router logits directly with a multiplicative correction that reduces all outgoing probabilities when \widehat\rho>\rho_{target} and increases them when it is below target. Apply the correction per parent type, optionally weighted by the Perron eigenvector so high-propagation types are penalized more strongly. For a depth-budgeted model choose \rho_{target}<1; for a recurrent model with deliberate long propagation use a schedule that warms from 0.8 to 1.0. The paper supplies the spectral interpretation and the criticality target; the neural implementation estimates \widehat M, \rho, r, and \ell empirically. First test on a 6-layer CIFAR-10 or WikiText-2 adaptive-depth Transformer against standard top-1/top-2 routing with equal average active FLOPs. Measure validation accuracy, active-token count by depth, maximum activation population, gradient norm, and loss variance. Success is fewer dead routes and lower variance, with equal or better accuracy at fixed FLOPs; a stronger signal is maintaining bounded active population over 20-50 recurrent steps while the baseline either collapses or explodes.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.