Degree-Corrected Hierarchical Router

Integrate this at the router or graph-pooling boundary, not inside the token MLP. For a graph model, let items be nodes and construct A from minibatch edge weights or exponentially averaged embedding similarities. For an MoE transformer, let items be token representatives, token clusters, or a bounded reservoir of vocabulary IDs; set A_ij to the exponentially averaged number of times i and j are sent to the same expert, co-occur in a local attention window, or have high cosine similarity. The router first computes ordinary logits z_i,e and soft assignments p_i,e = softmax(z_i,e). Build a sparse interaction graph from the top-k expert assignments, then recursively split each current group. Pseudocode: (1) initialize one group containing all items; (2) for every group, propose a binary split using the sign of the second eigenvector of the normalized interaction matrix or a two-means split of router embeddings; (3) estimate d_i = sum_j A_ij, m = (1/2)sum_i d_i, theta_i = d_i / sum_{j:g_j=g_i} d_j, and omega_ab = sum_{i in a,j in b} A_ij divided by the corresponding propensity normalization; (4) compute L_DCB, L_CM, and Delta; (5) commit the split only if Delta > 0 and both children exceed a minimum load; (6) repeat depth-first or breadth-first up to a fixed depth; (7) use the final groups as hierarchical routing masks or as initialization for learned expert logits. Estimate A, d_i, and m online with exponential moving averages, and compute the spectral split only every several hundred optimizer steps so routing overhead remains small. The paper supplies the degree-corrected-versus-configuration comparison and recursive refinement principle; the router-specific interaction graph and likelihood are the adaptation. Begin with a 4-layer, 8-expert Switch-style MoE on WikiText-103 or C4, preceded by a synthetic Zipf token-frequency benchmark. Compare ordinary load-balancing loss, flat frequency-normalized routing, and this method at equal FLOPs and capacity. Success means lower expert-load coefficient of variation, fewer collapsed experts, improved validation perplexity at the same active-parameter count, and partitions that remain stable across minibatches rather than tracking only high-frequency tokens.