# Control-Cost Laguerre MoE Routing

- ID: 618
- Canonical URL: https://synthcore.org/idea/618/control-cost-laguerre-moe-routing
- API JSON: https://synthcore.org/api/idea/618.json
- API Markdown: https://synthcore.org/api/idea/618.md
- Verification status: beats_baseline
- Source: [arXiv:2607.09139](https://arxiv.org/abs/2607.09139)
- Category: architecture
- Solves: scalability, accuracy, stability
- ML areas: moe-routing, transformer, embedding
- Math tags: optimal-transport, control-theory, geometry, optimization
- Ratings: usefulness 7/10; difficulty 6/10; novelty 6/10

## Idea description

Route tokens to experts using Laguerre cells defined by the minimum control energy needed to move a token embedding to each expert prototype, rather than by Euclidean distance or an unconstrained learned router logit. Per-expert dual weights deform the cells so that minibatch routing follows prescribed expert capacities, giving a geometrically interpretable alternative to auxiliary load-balancing losses.

## Mathematical statement

The paper's semi-discrete optimal transport problem is $\min_{T:\mathcal{X}\to\{\bm y_i\}_{i=1}^r,\,T_\#\mu=\nu}\int_{\mathcal X}c(\bm x,T(\bm x))\,d\mu$, where $\mu$ is the continuous source distribution of token embeddings, $\bm y_i$ are $r$ discrete target prototypes or experts, $c(x,y_i)$ is the transport ground cost, and $\nu=(\nu_1,\ldots,\nu_r)$ is the desired simplex of expert capacities. For a control-induced cost satisfying the twist condition, define the Control Laguerre cell $C_i(\psi)=\{x:c(x,y_i)-\psi_i\le c(x,y_j)-\psi_j\ \forall j\}$, where $\psi_i$ is a scalar dual weight. The routed map is $T(x)=y_i$ for $x\in C_i(\psi)$. A concrete minimum-energy linear-control cost for dynamics $\dot z=Az+Bu$ over horizon $\tau$ is $c_\tau(x,y)=\frac12(y-e^{A\tau}x)^\top W_\tau^{-1}(y-e^{A\tau}x)$, with controllability Gramian $W_\tau=\int_0^\tau e^{As}BB^\top e^{A^\top s}\,ds$. The capacity dual can be optimized with minibatch cell masses $\hat m_i=\frac1B\sum_{b=1}^B\mathbf 1[x_b\in C_i(\psi)]$ using $\psi_i\leftarrow\psi_i+\eta_\psi(\nu_i-\hat m_i)$; increasing $\psi_i$ enlarges expert $i$'s cell. The twist condition is the structural assumption that makes the cell assignment a valid almost-everywhere transport map rather than an arbitrary tie-breaking rule.

## Key formulas

- $$\underset{T:\mathcal{X}\rightarrow\{\bm{y}_{i}\}_{i=1}^{r}\mid T_{\#}\mu=\nu}{\mathrm{minimize}}\quad\int_{\mathcal{X}}c(\bm{x},T(\bm{x}))\,\mathrm d\mu$$
- $$C_i(\psi)=\left\{x\in\mathbb R^d:\ c(x,y_i)-\psi_i\le c(x,y_j)-\psi_j\ \text{for every }j\right\},\qquad T_\psi(x)=y_i\ \text{if }x\in C_i(\psi)$$
- $$c_\tau(x,y_i)=\frac12\bigl(y_i-e^{A\tau}x\bigr)^\top W_\tau^{-1}\bigl(y_i-e^{A\tau}x\bigr),\qquad W_\tau=\int_0^\tau e^{As}BB^\top e^{A^\top s}\,\mathrm ds$$
- $$\hat m_i=\frac1B\sum_{b=1}^{B}\mathbf 1\!\left[i=\arg\min_j\{c_\tau(x_b,y_j)-\psi_j\}\right],\qquad \psi_i\leftarrow\psi_i+\eta_\psi(\nu_i-\hat m_i)$$

## Implementation notes

(1) Integration point: replace the top-1 router in a sparse MoE layer. Let the preceding transformer block produce token vectors $h_b\in\mathbb R^d$. Add a projection $x_b=P h_b$ into a lower-dimensional routing space, maintain one prototype $y_i\in\mathbb R^d$ per expert, and compute the control cost to every expert. Use $A,B,\tau$ as fixed small matrices or learn them from a stable parameterization; precompute $F=e^{A\tau}$ and $G=W_\tau^{-1}$, so the cost is a batched quadratic form. (2) Pseudocode: initialize $\psi_i=0$ and capacities $\nu_i=1/r$; for each batch compute $q_b=F x_b$, $d_{bi}=0.5(y_i-q_b)^\top G(y_i-q_b)-\psi_i$; assign $a_b=\arg\min_i d_{bi}$; dispatch tokens by $a_b$; estimate $\hat m_i=B^{-1}\sum_b1[a_b=i]$; update $\psi_i\leftarrow\psi_i+\eta_\psi(\nu_i-\hat m_i)$ and optionally center $\psi\leftarrow\psi-\mathrm{mean}(\psi)$. Train the experts and projection with the ordinary task loss. Use a straight-through assignment or stop-gradient through routing. (3) Computed from the mathematics: $F$ and $G$ come from matrix exponential and controllability-Gramian calculations; cells and assignments use the exact Control Laguerre rule; only minibatch masses and learned prototypes are estimated or optimized empirically. If $W_\tau$ is ill-conditioned, use $(W_\tau+\epsilon I)^{-1}$. (4) First experiment: compare this router against Switch top-1 routing with its standard load-balancing loss on a 4- or 8-expert small Transformer trained on CIFAR-100 text labels, WikiText-2, or a synthetic mixture with clusters connected by anisotropic dynamics. Measure validation loss or perplexity, coefficient of variation of expert loads, dropped-token fraction, router overhead, and training stability across seeds. Success means lower load variance and fewer dropped tokens at equal capacity, with equal or better validation loss and no extra all-to-all communication.

## Verification

- Status: beats_baseline
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: helps
- Verdict: Built a minimum-energy Control-Cost Laguerre router with exact matrix-exponential dynamics, controllability Gramian costs, Laguerre assignments, and dual capacity updates. The mechanism manifested: numerical Gramian error was 5.3e-10, weaker second-coordinate actuation increased the measured cost ratio from 1.39 to 16.92, and dual ascent at eta=1.5 reduced maximum capacity error to 0.0018. In the synthetic capacity-drop comparison, Euclidean routing dropped 24.69% of tokens versus 4.23% for the dual Laguerre router, but no downstream neural task accuracy or wall-clock overhead win was tested.

### Mechanism check

- Verdict: Built a minimum-energy Control-Cost Laguerre router with exact matrix-exponential dynamics, controllability Gramian costs, Laguerre assignments, and dual capacity updates. The mechanism manifested: numerical Gramian error was 5.3e-10, weaker second-coordinate actuation increased the measured cost ratio from 1.39 to 16.92, and dual ascent at eta=1.5 reduced maximum capacity error to 0.0018. In the synthetic capacity-drop comparison, Euclidean routing dropped 24.69% of tokens versus 4.23% for the dual Laguerre router, but no downstream neural task accuracy or wall-clock overhead win was tested.
- Confidence: 8/10
- Limitations: Only a deterministic 2D synthetic mixture with four fixed prototypes was tested. There was no learned projection/prototype training, Transformer or language-model task, multi-seed analysis, GPU benchmark, communication measurement, or comparison with a learned Switch router and auxiliary load-balancing loss.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.0152
- Paired wins: 7/8
- Benchmark verdict: idea better (significant)

## Artifacts

- [README.md](https://synthcore.org/code/277/README.md)
- [bench_experiment.py](https://synthcore.org/code/277/bench_experiment.py)
- [bench_report.json](https://synthcore.org/code/277/bench_report.json)
- [experiment.py](https://synthcore.org/code/277/experiment.py)
- [report.md](https://synthcore.org/code/277/report.md)
- [report_bench_2026-08-31T001031.md](https://synthcore.org/code/277/report_bench_2026-08-31T001031.md)
- [results.json](https://synthcore.org/code/277/results.json)
- [Download all files as ZIP](https://synthcore.org/download/277)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.