Dyadic Expert-Overload Barrier

Integrate this at the MoE router, after top-k routing decisions or, preferably, using differentiable expected loads before dispatch. For a batch with router probabilities p[b,e], compute soft loads n_e=sum_b p[b,e]. For top-1 routing, use the straight-through hard assignment in the forward pass and p in the auxiliary loss. Set mean load n_bar=N/E, then normalize m_e=n_e/(n_bar+epsilon). Choose K=5 to 8 thresholds L_k=2^k with L_0=1, and coefficients q_k=q_0 2^(alpha k), using q_0=0.05 and alpha in [0,0.5]. An alternative is to estimate the 90th percentile of m_e during a warm-up window and set L_0 to half that value. Compute tail=sum_k q_k*relu(m-L_k) and add lambda*mean(tail) to the task loss. Compact pseudocode is: p=softmax(router_logits); n=sum_over_tokens(p); m=n/(N/E+eps); tail=0; for k in range(K): tail += q[k]*relu(m-L[k]); loss=task_loss + lambda*mean(tail); loss.backward(). The mathematical component is the convex hinge sum and its monotone superlinear slope; thresholds, coefficients, and lambda are engineering choices rather than paper-guaranteed values. First test a 4 to 8 expert Switch-Transformer-style MLP on C4 or a synthetic balanced classification task. Compare standard Switch load loss, squared coefficient-of-variation loss, and the dyadic penalty at equal auxiliary-loss weight. Record maximum and 99th-percentile expert load, dropped-token fraction, router entropy, task perplexity, gradient variance, and wall-clock throughput. Success means fewer overloaded or dropped tokens and smoother training at equal perplexity, or equal quality with a smaller auxiliary-loss coefficient. Reject the method if it merely forces uniform routing and reduces expert specialization.