# Relative-Entropy Routing for Expanding Experts

- ID: 2523
- Canonical URL: https://synthcore.org/idea/2523/relative-entropy-routing-for-expanding-experts
- API JSON: https://synthcore.org/api/idea/2523.json
- API Markdown: https://synthcore.org/api/idea/2523.md
- Verification status: unverified
- Source: [arXiv:2608.18426](https://arxiv.org/abs/2608.18426)
- Category: dynamics
- Solves: stability, generalization, sample-efficiency
- ML areas: moe-routing, optimizer, training-dynamics, regularization
- Math tags: dynamical-systems, information-theory, measure-theory, linear-algebra
- Ratings: usefulness 6/10; difficulty 6/10; novelty 7/10

## Idea description

Treat the router state as a symbolic base process and expert transformations as nonstationary expanding fiber maps. Add a relative entropy/free-energy constraint so that the router's conditional entropy is calibrated against the empirically measured growth rate of distinguishable expert trajectories, preventing premature expert collapse while retaining useful specialization.

## Mathematical statement

For a skew product $F(\omega,x)=(\sigma\omega,f_\omega(x))$, let $\sigma$ be an ergodic symbolic base with invariant measure $\nu$, and let $f_\omega$ be the fiber map selected by base sequence $\omega$. The relative variational principle identifies conditional fiber entropy with average fiberwise topological entropy: $h_\mu(F\mid\sigma)=\int h_{\mathrm{top}}(f_\omega)\,d\nu(\omega)$. In a mixture-of-experts network, $\omega_t$ is the router state, $p_t(i)=\Pr(I_t=i\mid\omega_t)$ is the expert distribution, and $H_t=-\sum_i p_t(i)\log p_t(i)$ is conditional routing entropy. The fiber growth rate is approximated by positive logarithmic Jacobian expansion. The adapted loss matches these two quantities while retaining a small entropy incentive.

## Key formulas

- $$P(\varphi)=\sup_{\mu\in\mathcal M(X,T)}\left[h_\mu(T)+\int_X\varphi\,d\mu\right],\qquad h_{\mathrm{top}}(T)=\sup_{\mu\in\mathcal M(X,T)}h_\mu(T)\quad\text{when }\varphi=0.$$
- $$F(\omega,x)=(\sigma\omega,f_\omega(x)),\qquad h_\mu(F\mid\sigma)=\int h_{\mathrm{top}}(f_\omega)\,d\nu(\omega).$$
- $$H_t=H(I_t\mid\omega_t)=-\sum_{i=1}^{K}p_t(i)\log p_t(i),\qquad \widehat h_{\mathrm{fiber},t}=\frac{1}{L}\sum_{\ell=0}^{L-1}\sum_{j=1}^{d}\max\left\{0,\log s_j\left(J_{e_{I_{t+\ell}}}(z_{t+\ell})\right)\right\},$$
- $$\mathcal L_{\mathrm{rel}}=\mathcal L_{\mathrm{task}}+\beta\left(\widehat h_{\mathrm{fiber}}-H_t\right)^2-\tau H_t,$$

## Implementation notes

Integrate the method into a top-1 or soft mixture-of-experts MLP, transformer feed-forward layer, or recurrent block. The router receives hidden state $z_t$, produces logits $a_t\in\mathbb R^K$, and forms $p_t=\operatorname{softmax}(a_t/T_r)$, where $T_r$ is router temperature. Experts $e_i$ map $z_t$ to the next hidden state. Maintain a window of length $L$ containing router probabilities, selected experts, and hidden states. Compute routing entropy $H_t=-\sum_i p_t(i)\log p_t(i)$. Estimate fiber growth using Jacobian-vector products: approximate the largest singular values $s_j(J_{e_i}(z))$ with one or two power iterations, retain only positive terms $\max(0,\log s_j)$, and average over the window. Form $L=L_{task}+\beta(\widehat h_{fiber}-H_t)^2-\tau H_t$ and update router and experts jointly. The paper contributes the relative entropy identity; Jacobian estimation and the neural penalty are adaptations. Start with $\tau>0$ during router warm-up, then cosine-decay $\tau$ while keeping $\beta$ fixed. First experiment: train on CIFAR-10 with identical experts, comparing standard Switch routing, entropy regularization, and this loss. Sweep expert spectral scale $g\in\{0.8,1.0,1.2,1.5\}$ and router temperature. The predicted signature is $H_t\approx\widehat h_{fiber}$ within 20% in the expanding regime. Collapse should increase sharply when $H_t-\widehat h_{fiber}<-0.2\log K$, and the measured branch-growth slope should agree with the positive-log-singular-value estimate within 20%.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.