# Cumulative-Fair MoE Capacity Envelopes

- ID: 2700
- Canonical URL: https://synthcore.org/idea/2700/cumulative-fair-moe-capacity-envelopes
- API JSON: https://synthcore.org/api/idea/2700.json
- API Markdown: https://synthcore.org/api/idea/2700.md
- Verification status: unverified
- Source: [arXiv:2608.23444](https://arxiv.org/abs/2608.23444)
- Category: training
- Solves: stability, scalability, accuracy
- ML areas: moe-routing, training-dynamics, scheduler
- Math tags: optimization, statistics, control-theory, dynamical-systems
- Ratings: usefulness 6/10; difficulty 5/10; novelty 5/10

## Idea description

Replace a static MoE load-balancing penalty with a two-stage capacity allocator. First compute each expert's technically feasible token capacity from latency, memory, and overflow constraints; then redistribute capacity using cumulative proportional fairness so experts that were repeatedly under-served receive more capacity later. Constrain the redistribution by an explicit efficiency budget, so fairness cannot silently cause an uncontrolled increase in routing loss or expert compute.

## Mathematical statement

For expert e in a set of E experts and training window t, let x_{e,t} be the number of tokens assigned to expert e, q^{tech}_{e,t} its technically feasible capacity, and d_{e,t} its unconstrained demand. Define cumulative served tokens S_{e,t}=\sum_{    au\le t}x_{e,    au} and cumulative demand D_{e,t}=\sum_{    au\le t}d_{e,    au}. The cumulative service ratio is r_{e,t}=S_{e,t}/(D_{e,t}+\epsilon), and the Jain index is J_t=(\sum_e r_{e,t})^2/[E\sum_e r_{e,t}^2]. The first-stage envelope satisfies 0\le q_{e,t}\le q^{tech}_{e,t} and \sum_e q_{e,t}\le C_t, where C_t is the available token or compute budget. The second stage chooses q^{fair}_{e,t} by minimizing deviation from the technical envelope while enforcing a disparity limit and an efficiency budget \Delta: \max_e r_{e,t}-\min_e r_{e,t}\le\rho and \sum_t\sum_e \ell_{e,t}(q^{fair}_{e,t})\le\sum_t\sum_e\ell_{e,t}(q^{tech}_{e,t})+\Delta. Here \ell_{e,t} is routing loss or excess compute, and fairness cost can be \Gamma_t=1-J_t.

## Key formulas

- $$J_t=\frac{\left(\sum_{e=1}^{E}r_{e,t}\right)^2}{E\sum_{e=1}^{E}r_{e,t}^2},\qquad r_{e,t}=\frac{S_{e,t}}{D_{e,t}+\epsilon},\qquad S_{e,t}=\sum_{\tau\le t}x_{e,\tau}.$$
- $$0\le q^{\mathrm{fair}}_{e,t}\le q^{\mathrm{tech}}_{e,t},\qquad \sum_{e=1}^{E}q^{\mathrm{fair}}_{e,t}\le C_t.$$
- $$\max_e r_{e,t}-\min_e r_{e,t}\le\rho,\qquad \sum_{t,e}\ell_{e,t}(q^{\mathrm{fair}}_{e,t})\le\sum_{t,e}\ell_{e,t}(q^{\mathrm{tech}}_{e,t})+\Delta.$$
- $$\min_{q^{\mathrm{fair}}}\ \sum_{t,e}w_{e,t}\left(q^{\mathrm{fair}}_{e,t}-q^{\mathrm{tech}}_{e,t}\right)^2+\lambda\sum_t(1-J_t).$$

## Implementation notes

Integrate the allocator immediately before top-k MoE routing. At the start of every window of B minibatches, estimate unconstrained expert demand d[e] from router probabilities and compute a technical envelope q_tech[e] using current expert memory and latency limits: q_tech[e]=min(capacity_e, floor(C*available_compute_e/sum_e available_compute_e)). Maintain cumulative demand D[e] and accepted tokens S[e] across windows. Solve the small quadratic projection problem for q_fair with projected gradient descent or a differentiable bisection multiplier: initialize q=q_tech; repeatedly update q toward the Jain or disparity objective; clip q to [0,q_tech]; project onto sum(q)<=C; and reject updates violating the efficiency budget Delta. During routing, assign at most q_fair[e] tokens to expert e; overflow tokens go to their next-best expert. After routing, update S using accepted assignments and measure routing loss, overflow, expert latency, and total FLOPs. The source-derived quantities are the Jain index, cumulative ratios, and two-stage feasibility/fairness construction. Estimate q_tech, the loss functions ell, and Delta empirically from expert latency, memory, overflow, and validation loss. First test a small Switch-Transformer or Mixtral-style model on C4, comparing standard top-k plus auxiliary load-balancing loss against this allocator at equal total FLOPs. Sweep rho and Delta. The quantitative prediction is that J_t rises toward one as rho is relaxed or Delta increases, while validation-loss or throughput degradation stays below Delta. When Delta is reduced below the minimum cost of equalization, the fairness constraint becomes active and produces a visible kink in J versus Delta. The allocator should make no change when the technical ratios already satisfy the disparity bound.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.