Relative-Entropy Routing for Expanding Experts

Integrate the method into a top-1 or soft mixture-of-experts MLP, transformer feed-forward layer, or recurrent block. The router receives hidden state $z_t$, produces logits $a_t\in\mathbb R^K$, and forms $p_t=\operatorname{softmax}(a_t/T_r)$, where $T_r$ is router temperature. Experts $e_i$ map $z_t$ to the next hidden state. Maintain a window of length $L$ containing router probabilities, selected experts, and hidden states. Compute routing entropy $H_t=-\sum_i p_t(i)\log p_t(i)$. Estimate fiber growth using Jacobian-vector products: approximate the largest singular values $s_j(J_{e_i}(z))$ with one or two power iterations, retain only positive terms $\max(0,\log s_j)$, and average over the window. Form $L=L_{task}+\beta(\widehat h_{fiber}-H_t)^2-\tau H_t$ and update router and experts jointly. The paper contributes the relative entropy identity; Jacobian estimation and the neural penalty are adaptations. Start with $\tau>0$ during router warm-up, then cosine-decay $\tau$ while keeping $\beta$ fixed. First experiment: train on CIFAR-10 with identical experts, comparing standard Switch routing, entropy regularization, and this loss. Sweep expert spectral scale $g\in\{0.8,1.0,1.2,1.5\}$ and router temperature. The predicted signature is $H_t\approx\widehat h_{fiber}$ within 20% in the expanding regime. Collapse should increase sharply when $H_t-\widehat h_{fiber}<-0.2\log K$, and the measured branch-growth slope should agree with the positive-log-singular-value estimate within 20%.