# Minimax-balanced progressive MoE splitting

- ID: 1994
- Canonical URL: https://synthcore.org/idea/1994/minimax-balanced-progressive-moe-splitting
- API JSON: https://synthcore.org/api/idea/1994.json
- API Markdown: https://synthcore.org/api/idea/1994.md
- Verification status: unverified
- Source: [arXiv:2608.08431](https://arxiv.org/abs/2608.08431)
- Category: architecture
- Solves: stability, scalability, accuracy
- ML areas: moe-routing, transformer, training
- Math tags: combinatorics, optimization, probability
- Ratings: usefulness 6/10; difficulty 5/10; novelty 8/10

## Idea description

Grow a mixture-of-experts layer by splitting one expert into two children while conserving its routing mass, and choose the split ratio to minimize the worst imbalance over all intermediate expert counts. Use the paper's sharp threshold as a hard design target: with n experts, some intermediate stage must have capacity ratio at least D_n = 2^{1-1/\lceil n/2\rceil}, so schedules substantially below this are impossible rather than merely difficult to discover. Initialize child router logits with log-probability offsets matching the split, while copying or lightly perturbing the parent's expert weights.

## Mathematical statement

For a partition or mass vector m = (m_1, ..., m_k) with every m_i > 0, define the discrepancy disc(m) = max_i m_i / min_i m_i. A binary refinement replaces one mass x by two positive masses ax and (1-a)x, where 0 < a < 1, and preserves total mass because ax + (1-a)x = x. A length-n strategy is a sequence of mass vectors m^(1), ..., m^(n), where m^(k) has k entries and each vector is obtained from the previous one by one such split. The paper proves the universal lower bound max_{1 <= k <= n} disc(m^(k)) >= D_n, with D_n = 2^{1-1/ceil(n/2)}, and states that the lex-merge strategy attains equality. Here m_i is an expert's routing prior or normalized capacity, k is the current number of experts, and a is the searched fraction assigned to the first child. The theorem supplies an optimality benchmark for the maximum intermediate load-prior ratio; the neural adaptation additionally regularizes observed token loads toward m_i.

## Key formulas

- $$\operatorname{disc}(\mathcal I)=\frac{\max\{|I|:I\in\mathcal I\}}{\min\{|I|:I\in\mathcal I\}}$$
- $$x\ \longmapsto\ ax,\,(1-a)x,\qquad 0<a<1,\qquad ax+(1-a)x=x$$
- $$\max_{1\le k\le n}\operatorname{disc}(m^{(k)})\ge 2^{1-1/\left\lceil n/2\right\rceil}=:D_n$$
- $$\mathcal L_{\mathrm{router}}=\mathcal L_{\mathrm{task}}+\lambda\sum_{i=1}^{k}\left(\widehat\ell_i-m_i\right)^2,\qquad \widehat\ell_i=\frac{1}{B}\sum_{b=1}^{B}p_i(x_b)$$

## Implementation notes

Integrate this into a Switch/Top-1 or Top-2 MoE router during progressive expert growth. Represent each expert by a positive routing prior m_i normalized so that the sum of all m_i is 1; use it as both a capacity target and an initialization offset in router logits. When growing from k to k+1 experts, choose an existing expert j and replace m_j by a m_j and (1-a)m_j. For a cheap MVP, enumerate a on the grid {0.05, 0.10, ..., 0.95}, form each candidate vector, and score it by the current discrepancy plus a look-ahead penalty for the next two planned splits. Select the candidate minimizing the maximum predicted discrepancy; optionally use beam search over the best eight partial schedules. The router computes logits z_i(x) = g_i(x) + log m_i, followed by softmax. After a split, initialize both child expert weights from the parent, set child biases to log a and log(1-a) relative to the parent's bias, and add small independent noise. During training, measure batch load \widehat{\ell}_i = B^{-1} sum_b p_i(x_b) and add the stated squared load-prior penalty, while retaining the normal task loss and capacity-overflow handling. The paper-derived quantities are D_n and each candidate prior discrepancy; empirical quantities are token loads, overflow rate, router entropy, and task loss. First test on a 4- or 8-expert Transformer MoE trained on CIFAR-100 or a small language-model corpus. Compare random/equal splitting, always splitting the largest expert, and the minimax schedule at equal parameters and FLOPs. Success means lower maximum expert overflow and fewer dead experts during growth, smoother loss immediately after each expansion, and no worse final perplexity or accuracy; a useful target is at least 20% lower peak load imbalance and fewer recovery steps after each split.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.