# Fast-Class MoE Router

- ID: 278
- Canonical URL: https://synthcore.org/idea/278/fast-class-moe-router
- API JSON: https://synthcore.org/api/idea/278.json
- API Markdown: https://synthcore.org/api/idea/278.md
- Verification status: mechanism_failed
- Source: [arXiv:2606.31596](https://arxiv.org/abs/2606.31596)
- Category: architecture
- Solves: stability, generalization, scalability
- ML areas: moe-routing, transformer, training
- Math tags: stochastic-processes, graph-theory, probability
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace a flat stochastic MoE router by a two-level continuous-time routing model: experts within a group communicate rapidly, while transitions between groups occur slowly. Use the effective class-level stationary distribution as a soft load-balancing prior, reducing routing oscillation while preserving expert specialization.

## Mathematical statement

The Markov chain tree theorem states that for a finite strongly connected directed graph with positive rates r_N(x,y), the invariant probability is \(\pi_N(x)=r_N(\mathcal{T}_x)/r_N(\mathcal{T})\), where \(\mathcal{T}_x\) is the set of directed spanning arborescences rooted at x, \(r_N(\mathcal{T}_x)=\sum_{\tau\in\mathcal{T}_x}\prod_{e\in\tau}r_N(e)\), and \(\mathcal{T}=\cup_x\mathcal{T}_x\). Partition experts into fast communicating classes \(M\in\mathcal{M}\). If \(\mu_M^N(x)\) is the invariant distribution of the fast chain inside class M and \(\widehat r_N\) is the trace-process rate after transient states are removed, the effective class-level rate is \(\bar r_N(M,M')=\sum_{x\in M}\mu_M^N(x)\sum_{y\in M'}\widehat r_N(x,y)\). In the neural adaptation, \(g(x)\) is the group of expert x, \(a_{xy}\) is a learned pairwise routing score, \(\tau\) is a temperature, and \(\gamma\gg1\) creates a fast within-group scale. The class stationary vector \(\bar\pi\) is computed from the effective generator and regularizes group occupancy.

## Key formulas

- $$\pi_N(x)=\frac{r_N(\mathcal{T}_x)}{r_N(\mathcal{T})},\qquad r_N(\mathcal{T}_x)=\sum_{\tau\in\mathcal{T}_x}\prod_{e\in\tau}r_N(e).$$
- $$\bar r_N(M,M')=\sum_{x\in M}\mu_M^N(x)\sum_{y\in M'}\widehat r_N(x,y),\qquad M\neq M'.$
- $$r_\gamma(x,y)=\begin{cases}\gamma\exp(a_{xy}/\tau),&g(x)=g(y),\ x\neq y,\\\exp(a_{xy}/\tau),&g(x)\neq g(y),\end{cases}$$
- $$\mathcal{L}_{\mathrm{stat}}=\mathrm{KL}(\rho\,\|\,\bar\pi)+\lambda\sum_M\left(\bar\pi(M)-\frac{1}{|\mathcal{M}|}\right)^2,$$

## Implementation notes

Integrate this into the MoE router before expert selection. Let each token representation h produce pairwise scores a_xy for a small candidate set of experts. Assign every expert to a group g(x), construct rates r_gamma using the displayed rule, and form a generator Q with Q_xy=r_gamma(x,y) for x != y and Q_xx=-sum_{y != x} r_gamma(x,y). For a cheap differentiable approximation, use P=I+Q/c with c >= max_x(-Q_xx), then run 10-30 power iterations v <- normalize(vP) to estimate the stationary vector. Separately form the fast within-group generators by retaining only same-group edges, estimate each mu_M with the same iteration, and compute bar_r(M,M') using the effective-rate formula. Run power iteration again on the class-level generator to obtain bar_pi. Add the displayed L_stat to the task loss, using rho as the minibatch group-load vector. During routing, use logits a_x + beta log(bar_pi(g(x))+epsilon), then select top-k experts subject to capacity. Stop-gradient through stationary estimates initially; later test implicit or unrolled differentiation. Begin with a 12-expert, 4-group, top-2 MoE Transformer on C4 or a synthetic mixture of token domains. Compare against Switch-style routing and its standard load-balancing loss at equal FLOPs. Record perplexity, expert-load coefficient of variation, token overflow, routing changes between successive batches, and all-to-all communication. Success is lower overflow and routing churn at matched perplexity, or faster loss decrease without increased communication.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a self-contained Fast-Class MoE router prototype with rate-scaled generators, stationary estimates, effective class-rate reduction, and a synthetic top-1 routing comparison. The Markov-chain tree theorem matched the stationary solve to 6.6e-15, and fast-class reduction error decreased from 2.09e-3 at gamma=10 to 2.56e-7 at gamma=1e5. However, the idea only marginally reduced churn while worsening overflow and load CV, so the promised practical win was not observed.

### Mechanism check

- Verdict: Built a self-contained Fast-Class MoE router prototype with rate-scaled generators, stationary estimates, effective class-rate reduction, and a synthetic top-1 routing comparison. The Markov-chain tree theorem matched the stationary solve to 6.6e-15, and fast-class reduction error decreased from 2.09e-3 at gamma=10 to 2.56e-7 at gamma=1e5. However, the idea only marginally reduced churn while worsening overflow and load CV, so the promised practical win was not observed.
- Confidence: 9/10
- Limitations: Only a small synthetic top-1 router was tested; no learned pairwise router, top-2 capacity-constrained dispatch, backpropagation, transformer/C4 training, equal-FLOP perplexity comparison, communication measurement, or differentiable stationary estimation was evaluated.

## Artifacts

- [experiment.py](https://synthcore.org/code/1249/experiment.py)
- [report.md](https://synthcore.org/code/1249/report.md)
- [results.json](https://synthcore.org/code/1249/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1249)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.