Strongly Connected Sparse Routing

Integrate this into the MoE router immediately after router logits and before top-k dispatch. Use overlapping contexts such as token windows of length 2 or 4, consecutive microbatch groups, or the same token under two data augmentations. For each context c, compute router probabilities p_c over expert assignments; define the Boolean support F_p(c)={e:p_c(e)>epsilon}, with epsilon initially 1e-3 and optionally annealed downward. Build a bipartite incidence graph whose nodes are maximal contexts and supported assignments; connect contexts when their overlap restriction maps are bijections on the currently retained supports. Pseudocode: (1) logits z_c = router(h_c); (2) p_c=softmax(z_c/T); (3) retain edges e with p_c(e)>epsilon; (4) repeatedly remove an edge only if every restricted child context remains nonempty and every parent-to-child restriction remains surjective; (5) compute connected components among maximal contexts after keeping only bijective overlap links; (6) add lambda times the number of disconnected component pairs to the loss, or use a straight-through support mask; (7) renormalize p_c on retained experts and dispatch top-k. The paper supplies the exact support-order and connectivity certificate; the engineer estimates supports from probabilities and connectivity with union-find plus degree checks. First test Switch-Transformer or a 4-expert MLP-MoE on CIFAR-100 or a small C4 language subset, comparing standard top-2 routing at equal FLOPs. Success means lower router entropy and expert collapse, more balanced expert utilization, and equal-or-better validation loss with fewer active expert-token edges.