# Fermi-Dirac Capacity Projection for MoE Routing

- ID: 3111
- Canonical URL: https://synthcore.org/idea/3111/fermi-dirac-capacity-projection-for-moe-routing
- API JSON: https://synthcore.org/api/idea/3111.json
- API Markdown: https://synthcore.org/api/idea/3111.md
- Verification status: unverified
- Source: [arXiv:2609.03552](https://arxiv.org/abs/2609.03552)
- Category: architecture
- Solves: stability, accuracy, scalability
- ML areas: moe-routing, transformer, training
- Math tags: convex-analysis, information-theory, optimization
- Ratings: usefulness 6/10; difficulty 6/10; novelty 6/10

## Idea description

Replace heuristic softmax load balancing with a convex entropic projection that computes expert assignment fractions satisfying explicit linear capacity and load constraints. The projection is bounded componentwise and keeps poorly identified routing decisions away from extreme 0/1 allocations, reducing expert collapse and overload under noisy or shifted batches.

## Mathematical statement

The paper considers the bounded linear inverse problem Aξ=y with ξ∈K=∏_j[a_j,b_j], where A∈R^{M×N} is a constraint matrix, y∈R^M is the desired constraint vector, and ξ∈R^N is the unknown allocation. It selects the unique minimizer of the separable Fermi-Dirac negative entropy Ψ(ξ)=Σ_j[p_j ln(p_j)+(1-p_j)ln(1-p_j)], with p_j=(ξ_j-a_j)/(b_j-a_j). Each p_j∈[0,1] is a normalized coordinate, and strict convexity gives uniqueness in the feasible set. For equality constraints, the KKT equations give ξ_j(λ)=a_j+(b_j-a_j)σ(-(b_j-a_j)(A^Tλ)_j), where λ∈R^M is the dual multiplier and σ(u)=1/(1+e^{-u}) is the logistic function. The multiplier is chosen so that Aξ(λ)=y, so box constraints are enforced automatically while optimization is reduced to an M-dimensional nonlinear residual equation.

## Key formulas

- $$A\xi=y,\qquad \xi\in\mathcal{K}=\prod_{j=1}^{N}[a_j,b_j].$$
- $$\Psi(\xi)=\sum_{j=1}^{N}\left[p_j\ln(p_j)+(1-p_j)\ln(1-p_j)\right],\qquad p_j=\frac{\xi_j-a_j}{b_j-a_j}.$$
- $$\xi_j(\lambda)=a_j+(b_j-a_j)\,\sigma\left(-(b_j-a_j)(A^T\lambda)_j\right),\qquad \sigma(u)=\frac{1}{1+e^{-u}}.$$
- $$r(\lambda)=A\xi(\lambda)-y=0,\qquad \min_{a\leq\xi\leq b}\frac{1}{2}\|A\xi-y\|_2^2+\tau\Psi(\xi).$$

## Implementation notes

Integrate this as a batch-level routing projection after the ordinary router produces token-to-expert logits and before dispatch. For a batch of T tokens and E experts, create a vector x containing fractional assignments x_{t,e}. Set lower bounds a_{t,e}=0 and upper bounds b_{t,e}=1, or use b_{t,e}=c_e/T when expert e has capacity c_e. Build A and y from linear constraints: every token receives total mass one, so Σ_e x_{t,e}=1; every expert receives a requested load l_e, so Σ_t x_{t,e}=l_e; additional rows can enforce per-domain or per-modality quotas. Compute router logits z_{t,e} with the usual gate and solve min_x[-<z,x>+τΨ(x)] subject to Ax=y and a≤x≤b. Let d_j=b_j-a_j. The KKT coordinate update is x_j(λ)=a_j+d_jσ((z_j-d_j(A^Tλ)_j)/τ). Solve r(λ)=Ax(λ)-y=0 with 10 to 30 damped Newton iterations. The Jacobian is J(λ)=-A diag(d_j^2σ_j(1-σ_j)/τ) A^T; add εI with ε=1e-5 for conditioning, use a line search, and stop when the infinity norm of r is below 1e-4. Initially detach the projection or backpropagate through the unrolled iterations in a second version. Estimate l_e from uniform desired loads or an exponential moving average of recent router probabilities. Test on an 8-expert Switch-Transformer-style model trained on a small C4 or Wikitext-103 subset. Compare standard top-1 routing plus auxiliary load loss against entropic projection followed by top-1 or top-2 dispatch. Record expert overflow, dropped-token rate, routing entropy, validation perplexity, projection runtime, and total training throughput. The method succeeds if it reduces overflow and token dropping at equal perplexity, or improves perplexity at the same capacity factor without router-collapse spikes under deliberately shifted or corrupted batches.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.