# Garding Geometric-Mean Load Balancer

- ID: 620
- Canonical URL: https://synthcore.org/idea/620/garding-geometric-mean-load-balancer
- API JSON: https://synthcore.org/api/idea/620.json
- API Markdown: https://synthcore.org/api/idea/620.md
- Verification status: mechanism_failed
- Source: [arXiv:2607.09168](https://arxiv.org/abs/2607.09168)
- Category: regularization
- Solves: generalization, stability
- ML areas: moe-routing, loss, regularization
- Math tags: convex-analysis, geometry, complex-analysis
- Ratings: usefulness 6/10; difficulty 3/10; novelty 7/10

## Idea description

Replace or augment entropy-based MoE load balancing with a structured concave utility over expert loads. The utility is the geometric mean of positive linear coverage factors, so it rewards underused directions strongly while exhibiting diminishing returns for already-covered directions. Positive coefficients can encode expert capacity, hardware placement, or expert groups.

## Mathematical statement

A posynomial is defined by $$p(x)=\sum_{a\in\mathcal{A}}c_a x^a=\sum_{a\in\mathcal{A}}c_a\prod_{i=1}^n x_i^{a_i},\qquad c_a>0,\quad a\in\mathbb{R}_{\geq 0}^n.$$ It is homogeneous of degree d when every exponent vector satisfies $$\sum_{i=1}^n a_i=d.$$ Define $$\Gamma_{\alpha}=\{re^{i\theta}\mid r>0,\ |\theta|<\alpha\pi/2\}.$$ A posynomial is \(\Gamma_\alpha\)-stable when its principal branch has no zero on \(\Gamma_\alpha^n\). The theorem states that homogeneous \(\Gamma_1\)-stable posynomials have a concave degree-normalized root \(p(x)^{1/d}\); more generally, \(\Gamma_\alpha\)-stability yields concavity of \(p(x)^{\alpha/d}\). For an implementable special case, choose positive coefficient vectors \(c_j\in\mathbb{R}_{>0}^E\) and define $$p(\ell)=\prod_{j=1}^{d}\left(\sum_{e=1}^{E}c_{je}\ell_e\right).$$ Each factor is a positive linear form and is zero-free when all inputs lie in the right half-plane, so the product is a degree-d stable posynomial. Therefore $$U(\ell)=p(\ell)^{1/d}=\left[\prod_{j=1}^{d}\left(\sum_{e=1}^{E}c_{je}\ell_e\right)\right]^{1/d}$$ is concave for positive expert loads \(\ell_e\).

## Key formulas

- $$p(x)=\sum_{a\in\mathcal{A}}c_a x^a=\sum_{a\in\mathcal{A}}c_a\prod_{i=1}^n x_i^{a_i},\qquad c_a>0,\quad a\in\mathbb{R}_{\geq 0}^n.$$
- $$\Gamma_{\alpha}=\{re^{i\theta}\mid r>0,\ |\theta|<\alpha\pi/2\},\qquad p(z)\neq 0\ \text{for all }z\in\Gamma_{\alpha}^n.$$
- $$p\text{ homogeneous of degree }d\text{ and }\Gamma_1\text{-stable}\quad\Longrightarrow\quad p(x)^{1/d}\text{ is concave on }\mathbb{R}_{>0}^n.$$
- $$U(\ell)=\left[\prod_{j=1}^{d}\left(\sum_{e=1}^{E}c_{je}\ell_e\right)\right]^{1/d},\qquad \mathcal{L}_{\mathrm{balance}}=-\lambda U(\ell).$$

## Implementation notes

(1) Integration point: modify the auxiliary load-balancing loss of a sparse or dense Mixture-of-Experts router. For a batch of T tokens, let \(r_{te}\) be the differentiable router probability assigned to expert e by token t, and define the normalized soft load \(\ell_e=T^{-1}\sum_{t=1}^{T}r_{te}\). Keep top-k dispatch and capacity handling unchanged, but replace the standard entropy or Switch-style balancing term with \(-\lambda U(\ell)\), or use a weighted sum of the two during an ablation.

(2) Pseudocode:
```text
logits = router(hidden_states)
r = softmax(logits, dim=expert)
load[e] = mean(r[:, e])
for j in 1..d:
    factor[j] = sum_e(c[j, e] * load[e])
logU = mean_j(log(factor[j] + eps))
U = exp(logU)
loss = task_loss - lambda * U
backprop(loss)
```
Use log-space evaluation because \(\log U=d^{-1}\sum_j\log(\sum_e c_{je}\ell_e)\) avoids overflow and underflow. Choose \(c_{je}=1\) for a symmetric baseline. For capacity-aware routing, set \(c_{je}\) larger for experts with greater capacity; for group balancing, use several vectors with different positive group weights. If minibatch loads are noisy, maintain an exponential moving average \(\bar{\ell}_t=\beta\bar{\ell}_{t-1}+(1-\beta)\ell_t\) and compute the auxiliary utility from \(\bar{\ell}_t\), while gradients continue through the current batch load.

(3) The paper supplies the concavity and stability guarantee for the chosen positive linear-factor construction. The coefficients, number of factors d, regularization strength \(\lambda\), smoothing constant eps, and EMA coefficient \(\beta\) are empirical design parameters. Monitor the gradient of the auxiliary term separately from the task gradient.

(4) First cheap experiment: train a small Switch-Transformer-style language model with 8 experts on WikiText-103 or a small C4 subset. Compare standard Switch balancing, router entropy regularization, and this utility at matched auxiliary-loss magnitude and identical model/FLOP budgets. Sweep \(d\in\{2,4,8\}\) and \(\lambda\). Record validation perplexity, expert-load coefficient of variation, dropped-token rate, router-gradient variance, and throughput. The method succeeds if it lowers load variance and dropped-token rate without increasing perplexity or reducing useful routing specialization; a stronger result is lower perplexity at equal expert utilization and smoother load trajectories than entropy regularization.

## Verification

- Status: mechanism_failed
- Mechanism evidence: yes
- Mechanism confirmed: no
- Verdict: Built a log-space geometric-mean load-balancing utility and a fixed-seed toy MoE router experiment. The mathematical claim was verified: 1,000 Jensen tests showed no concavity violation, and the Hessian had maximum eigenvalue approximately -1.1e-16. The structured GM variant reduced soft-load CV to 1.351 versus 1.617 for Switch-style balancing, but increased task loss to 0.277 versus 0.176 and did not improve hard routing imbalance or capacity-1 drop rate. The literal symmetric c=1 version is constant on the normalized-load simplex and supplies no balancing gradient there, so this MVP does not demonstrate a practical win.

### Mechanism check

- Verdict: Built a log-space geometric-mean load-balancing utility and a fixed-seed toy MoE router experiment. The mathematical claim was verified: 1,000 Jensen tests showed no concavity violation, and the Hessian had maximum eigenvalue approximately -1.1e-16. The structured GM variant reduced soft-load CV to 1.351 versus 1.617 for Switch-style balancing, but increased task loss to 0.277 versus 0.176 and did not improve hard routing imbalance or capacity-1 drop rate. The literal symmetric c=1 version is constant on the normalized-load simplex and supplies no balancing gradient there, so this MVP does not demonstrate a practical win.
- Confidence: 8/10
- Limitations: This is a small differentiable per-token router toy rather than a language model with learned hidden states, top-k dispatch, EMA loads, or realistic capacity handling. Coefficients and auxiliary weights were not swept, and hard argmax routing remained concentrated on the two task-preferred experts, so the structured utility's soft-load improvement did not translate to dispatch-level improvement.

## Artifacts

- [experiment.py](https://synthcore.org/code/1415/experiment.py)
- [report.md](https://synthcore.org/code/1415/report.md)
- [results.json](https://synthcore.org/code/1415/results.json)
- [Download all files as ZIP](https://synthcore.org/download/1415)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.