# Laplace-Heterogeneous MoE Routing

- ID: 46
- Canonical URL: https://synthcore.org/idea/46/laplace-heterogeneous-moe-routing
- API JSON: https://synthcore.org/api/idea/46.json
- API Markdown: https://synthcore.org/api/idea/46.md
- Verification status: failed_benchmark
- Source: [arXiv:2608.22122](https://arxiv.org/abs/2608.22122)
- Category: architecture
- Solves: scalability, stability, accuracy
- ML areas: moe-routing, transformer
- Math tags: probability, statistics, dynamical-systems
- Ratings: usefulness 6/10; difficulty 4/10; novelty 6/10

## Idea description

Replace the usual hand-designed expert-load penalty with a heterogeneous survival penalty derived from a susceptibility distribution. Each expert receives an availability factor q_e=G(A_e), where A_e is its cumulative recent routing pressure and G_e is a learned or fixed mixture of exponentials; highly used experts are suppressed smoothly, while heterogeneous experts can have different resistance to pressure. The mixture produces adaptive curvature and long-tailed penalties that may reduce router collapse and expert overflow without adding auxiliary loss terms.

## Mathematical statement

The paper defines a nonnegative susceptibility random variable Lambda with law F and Laplace transform G(a)=E[exp(-a Lambda)]=integral exp(-a lambda) dF(lambda). The susceptible fraction under cumulative pressure a is s(a)=s_0 G(a). Its effective susceptibility is the exponentially tilted mean m(a)=-G'(a)/G(a)=E[Lambda exp(-a Lambda)]/E[exp(-a Lambda)], where the tilted distribution weights susceptibility lambda proportionally to exp(-a lambda). For finite second moment, m'(a)=-Var_a(Lambda)<=0, so the pressure response weakens monotonically. In the neural adaptation, expert e has nonnegative pressure A_e, availability q_e=G(A_e), and routing logit z_e+log(q_e+epsilon). Use a finite mixture G(A_e)=sum_{k=1}^K w_k exp(-lambda_k A_e), with w_k>=0, sum_k w_k=1, and lambda_k>=0. The adaptive hazard is m(A_e)=sum_k pi_k(A_e) lambda_k, where pi_k(A_e)=w_k exp(-lambda_k A_e)/G(A_e).

## Key formulas

- $$G(a)=\int_{[0,\infty)}e^{-a\lambda}\,F(d\lambda)=\mathbb{E}[e^{-a\Lambda}],\qquad s(t)=s_0G(A(t)),\qquad A(t)=\tau\int_0^t\iota(u)\,du.$$
- $$\overline{\lambda}(A)=-\frac{G'(A)}{G(A)}=\frac{\mathbb{E}[\Lambda e^{-A\Lambda}]}{\mathbb{E}[e^{-A\Lambda}]}.$$
- $$G(A)=\sum_{k=1}^K w_k e^{-\lambda_kA},\qquad \pi_k(A)=\frac{w_ke^{-\lambda_kA}}{G(A)},\qquad m(A)=\sum_k\pi_k(A)\lambda_k.$$
- $$m'(A)=-\operatorname{Var}_{A}(\Lambda)=-\left[\sum_k\pi_k(A)\lambda_k^2-m(A)^2\right]\le 0.$$

## Implementation notes

Integrate the mechanism into the router immediately before top-k expert selection. For a batch of tokens x_i, let r_ie=x_i^T v_e be the ordinary router logit and let p_e be an exponentially averaged expert-pressure state. First compute soft routing mass u_e=(1/B) sum_i softmax(r_i)_e, then update pressure with p_e <- beta p_e+(1-beta)u_e; use stop-gradient through p_e for the first experiment so the router cannot exploit state-update dynamics. Define nonnegative susceptibility values by lambda_k=softplus(a_k), mixture weights w_k=softmax(b)_k, and G(p_e)=sum_k w_k exp(-lambda_k p_e). Modify logits as r'_ie=r_ie+alpha log(G(p_e)+epsilon), then select top-k experts using r'. The pseudocode is: compute r; compute u from the unmodified softmax; update p; compute q_e=sum_k w_k exp(-lambda_k p_e); compute r'=r+alpha log(q_e+epsilon); select top-k from r'; dispatch and train normally. Initially fix lambda_k on a logarithmic grid, such as {0.25,1,4,16}, and use uniform w_k; optionally learn lambda_k and w_k with a weak entropy penalty on w and a constraint on the average lambda scale. The paper's mathematical guarantee is monotone decreasing effective hazard m'(p)<=0; verify this numerically during training. Monitor expert-load coefficient of variation, token overflow rate, router entropy, validation loss, and throughput. Start with a 6-layer Transformer on Wikitext-103 or a synthetic language task, using 8 experts and top-2 routing. Compare against standard top-2 routing with a Switch-style load-balancing loss at equal capacity and FLOPs. Success means lower overflow and utilization variance at matched perplexity, or improved perplexity after removing the auxiliary balancing loss. Include ablations for a single exponential, the mixture, and a fixed linear load penalty.

## Verification

- Status: failed_benchmark
- Mechanism evidence: yes
- Mechanism confirmed: no
- Practical verdict: no_effect
- Verdict: Built a reusable NumPy Laplace-heterogeneous MoE router with EMA pressure, finite-mixture availability, effective hazard, and top-k selection. The mechanism checks were confirmed: m'(A)=-Var_A(lambda) had max error 2.65e-8, q'/q=-m had error 4.12e-7, and the observed 1% asymptotic crossover pressure 7.65 closely matched the predicted 7.60. In the deterministic routing stress test, the mixture reduced load CV from 2.397 to 0.180 and overflow from 0.777 to 0.014 versus baseline, but this does not establish language-model quality improvement.

### Mechanism check

- Verdict: Built a reusable NumPy Laplace-heterogeneous MoE router with EMA pressure, finite-mixture availability, effective hazard, and top-k selection. The mechanism checks were confirmed: m'(A)=-Var_A(lambda) had max error 2.65e-8, q'/q=-m had error 4.12e-7, and the observed 1% asymptotic crossover pressure 7.65 closely matched the predicted 7.60. In the deterministic routing stress test, the mixture reduced load CV from 2.397 to 0.180 and overflow from 0.777 to 0.014 versus baseline, but this does not establish language-model quality improvement.
- Confidence: 9/10
- Limitations: Only a deterministic NumPy toy routing stress test was run; no trained Transformer, language-model loss, perplexity, CUDA benchmark, learned mixture parameters, top-2 dispatch capacity implementation, or equal-FLOP training comparison was tested.

### Practical benchmark

- Paired seeds: 8
- Baseline mean: 0
- Idea mean: 0
- p-value: 0.8799
- Paired wins: 3/8
- Benchmark verdict: no measurable effect

## Artifacts

- [bench_report.json](https://synthcore.org/code/693/bench_report.json)
- [experiment.py](https://synthcore.org/code/693/experiment.py)
- [laplace_moe.py](https://synthcore.org/code/693/laplace_moe.py)
- [report.md](https://synthcore.org/code/693/report.md)
- [report_bench_2026-08-31T223231.md](https://synthcore.org/code/693/report_bench_2026-08-31T223231.md)
- [results.json](https://synthcore.org/code/693/results.json)
- [stage2_bench.py](https://synthcore.org/code/693/stage2_bench.py)
- [Download all files as ZIP](https://synthcore.org/download/693)

## Disclaimer

AI-generated research hypothesis, automatically tested. Not peer-reviewed.