# Layered simplex sparse router

- ID: 2594
- Canonical URL: https://synthcore.org/idea/2594/layered-simplex-sparse-router
- API JSON: https://synthcore.org/api/idea/2594.json
- API Markdown: https://synthcore.org/api/idea/2594.md
- Verification status: unverified
- Source: [arXiv:2608.19908](https://arxiv.org/abs/2608.19908)
- Category: architecture
- Solves: scalability, generalization, accuracy
- ML areas: moe-routing, transformer, regularization
- Math tags: probability, information-theory, statistics
- Ratings: usefulness 6/10; difficulty 5/10; novelty 7/10

## Idea description

Replace an MoE router's single softmax distribution with a normalized coordinate-wise product of several simplex-valued routing factors. The product preserves positivity and normalization but, as depth grows, concentrates mass on a small subset of experts, creating a mathematically controlled heavy-tailed routing prior rather than relying only on an auxiliary load-balancing loss.

## Mathematical statement

The LSA construction samples independent uniform simplex vectors U_1,...,U_L in the simplex \(\Delta_E=\{u\in\mathbb{R}_+^E:\sum_{e=1}^E u_e=1\}\), forms unnormalized coordinates \(Y_e=\prod_{\ell=1}^L U_{\ell,e}\), and normalizes them as \(\theta_e=Y_e/\sum_{j=1}^E Y_j\). Here E is the number of experts, L is the multiplicative depth, and \(\theta\) is the router probability vector. For a uniform simplex coordinate, \(\log U_{\ell,e}\) has mean \(-\gamma\), where \(\gamma\) is the Euler–Mascheroni constant, and variance \(\pi^2/6\). Therefore \(\log Y_e=\sum_{\ell=1}^L\log U_{\ell,e}\) has mean \(-\gamma L\) and variance \(L\pi^2/6\); increasing L spreads coordinate scales exponentially and produces sparse, heavy-tailed normalized vectors. The associated Bayesian sequence mixture is \(Q_N(x^N)=\mathbb{E}_{\theta}\prod_{t=1}^N\theta_{x_t}\), and its regret is controlled by negative log prior mass near the target, \(-\log w(\Theta_0^\epsilon)\). In the neural adaptation, the random prior is replaced by trainable factor routers \(u_\ell(h)\in\Delta_E\), while the same product-normalization and depth-dependent concentration are retained.

## Key formulas

- $$U_1,\ldots,U_L\overset{\mathrm{iid}}{\sim}\operatorname{Unif}(\Delta_E),\qquad Y_e=\prod_{\ell=1}^{L}U_{\ell,e},\qquad \theta_e=\frac{Y_e}{\sum_{j=1}^{E}Y_j}.$$
- $$\mathbb{E}[\log U_{\ell,e}]=-\gamma,\qquad \operatorname{Var}(\log U_{\ell,e})=\frac{\pi^2}{6},\qquad \log Y_e\approx\mathcal{N}\!\left(-\gamma L,\frac{\pi^2L}{6}\right).$$
- $$L=\operatorname{round}(c\ln E),\qquad c>0,$$
- $$Q_N(x^N)=\mathbb{E}_{\theta\sim\mathcal{M}_{E,L}}\prod_{t=1}^{N}\theta_{x_t},\qquad \operatorname{regret}\ \text{is governed by}\ -\log w(\Theta_0^{\epsilon}).$$

## Implementation notes

Integrate this at the token-to-expert router in a standard top-k MoE Transformer. Let h be a token hidden state and let g_l(h) be the logits of factor router l, each of dimension E. Compute u_l=softmax(g_l/T_l), where T_l is a temperature, then compute log_p[e]=sum_l log(u_l[e]+epsilon), p=softmax(log_p), and select the top-k experts using p. This is the paper's product construction in log space, with trainable simplex factors replacing the random uniform vectors. Initialize each g_l so that u_l is close to a random uniform-simplex draw: sample z_l[e] independently from Exp(1), set u_l=z_l/sum_e z_l[e], and initialize g_l=log(u_l). Use L=round(c log E) as the primary setting, plus L in {1,2,4,8} for an ablation. To prevent permanent expert collapse, add only a weak load penalty based on the squared difference between batch mean routing probabilities and 1/E; avoid a strong balancing coefficient initially. During training, compute the product in log space and stop gradients only through the top-k indices, not through p. First test a 6-layer, 8-expert Switch-style Transformer on WikiText-103 or a small C4 subset, with fixed activated-expert FLOPs. Compare ordinary softmax top-1 routing, product routing at L=2,4,8, and logarithmic depth. Measure validation perplexity, expert utilization entropy, effective support exp(-sum_e p_e log p_e), top-1 routing mass, token overflow, and wall-clock throughput. Success means lower perplexity at equal activated FLOPs, fewer overflowed tokens, or lower effective support without worse perplexity; failure is increased collapse or no improvement after temperature tuning.

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.