# Orthogonal Latin Expert Router

- ID: 714
- Canonical URL: https://synthcore.org/idea/714/orthogonal-latin-expert-router
- API JSON: https://synthcore.org/api/idea/714.json
- API Markdown: https://synthcore.org/api/idea/714.md
- Verification status: unverified
- Source: [arXiv:2607.10890](https://arxiv.org/abs/2607.10890)
- Category: architecture
- Solves: stability, scalability, speedup
- ML areas: moe-routing, mlp, batching, training-dynamics
- Math tags: combinatorics, graph-theory, algebra
- Ratings: usefulness 6/10; difficulty 5/10; novelty 6/10

## Idea description

Use mutually orthogonal Latin labels as deterministic expert assignments for token batches. Each routing family is individually balanced, and pairs of families avoid repeated co-assignment patterns, enabling multiple routing rounds or auxiliary experts without the severe load collisions caused by independent random hashing.

## Mathematical statement

For a batch arranged as an n by n grid, let L_1 through L_h be Latin squares of order n. Latinness means each expert label u in [n] occurs exactly once in every row and column, hence exactly n times in the full n squared token batch. Mutual orthogonality means that for a and b distinct, every ordered label pair (u,v) occurs exactly once: the number of positions (i,j) satisfying L_a(i,j)=u and L_b(i,j)=v is 1. The first identity gives exact capacity balance for each routing family; the second gives pairwise diversity when two routing families are used sequentially or as primary and backup experts. The adapted router uses a learned score only to choose among a small set of Latin-prescribed candidate labels, rather than allowing arbitrary top-1 assignments.

## Key formulas

- $$\forall i,j\in[n],\quad \{L_s(i,1),\ldots,L_s(i,n)\}=[n],\qquad \{L_s(1,j),\ldots,L_s(n,j)\}=[n].$$
- $$L_a\perp L_b\quad\Longrightarrow\quad \#\{(i,j)\in[n]^2:L_a(i,j)=u,\ L_b(i,j)=v\}=1\quad\forall u,v\in[n].$$
- $$p_s(u)=\frac{1}{n^2}\sum_{i=1}^{n}\sum_{j=1}^{n}\mathbf{1}\{L_s(i,j)=u\}=\frac1n,$$
- $$e_t=\arg\max_{u\in\mathcal{C}_t}\left(z_{t,u}-\tau\,\log\big(\widehat{q}_u+\epsilon\big)\right),\qquad \mathcal{C}_t=\{L_s(i,j):s\in S_t\}.$$

## Implementation notes

Integrate this at the MoE router immediately before top-1 or top-2 expert selection. Choose E=n experts and process each microbatch in chunks of n squared tokens; map token position t to grid coordinates (i,j). Precompute h mutually orthogonal Latin squares L_s, or use L_a(i,j)=((i-1)+a(j-1)) modulo n plus 1 for distinct slopes when n is prime. For token t=(i,j), form a candidate set C_t from the labels L_s(i,j) over a small subset S_t of routing families. The router network emits logits z_t,u for candidate experts only. Maintain an exponential moving average qhat_u of the number of tokens sent to expert u; select argmax of z_t,u minus tau times log(qhat_u plus epsilon), then dispatch by expert. For top-2 routing, choose the best two distinct candidates and renormalize their softmax weights. The Latin tables and their balance and orthogonality guarantees are exact; qhat_u, logits, and temperature tau are empirical. First test a 4-expert or 8-expert Switch-style MLP on a small language-model benchmark such as WikiText-2, comparing standard noisy top-1 routing, random hash routing, and this candidate-constrained router at equal expert FLOPs. Record dropped-token fraction, coefficient of variation of expert loads, auxiliary load-balancing loss, training loss at fixed steps, and all-reduce or dispatch time. Success means near-zero overflow or dropped tokens, lower load variance, and equal or better perplexity without increasing router FLOPs; a useful systems signal is reduced padding in expert GEMMs and faster step time at the same parameter count.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.