Jumbled Router Certificate

Integrate this at the MoE router immediately after computing router probabilities and before top-1 or top-2 dispatch. Use a batch of N tokens, E experts, t token groups, and r expert groups; contiguous token groups test positional locality, while a fresh random token permutation on each diagnostic step prevents the router from overfitting fixed groups. Aggregate experts into r groups so the block matrix is small. Pseudocode: (1) compute router logits z[n,e] and probabilities p[n,e]=softmax(z[n,:]); (2) choose hard assignments h[n]=argmax_e p[n,e] for a certificate, while retaining p for gradients; (3) form A[i,j]=sum over n in token_group_i and e in expert_group_j of p[n,e]; (4) set q[i,j]=|I_i| times the target mass of expert group j; (5) set m=N/t and lambda=beta*m/sqrt(N), with beta from the displayed formula and k treated as a tunable pattern-complexity proxy; (6) add alpha times L_jumbled to the task loss. For hard routing, replace A with integer counts and report max_{i,j}|A[i,j]-q[i,j]|/(lambda+delta). The paper supplies the hypergeometric baseline and concentration scale; the soft-routing extension, alpha, t, and k are empirical choices. Start with a 4-layer, 8-expert Transformer on WikiText-103 or a synthetic long-range token task. Compare standard Switch-style global load balancing against global balancing plus this term at equal FLOPs. Measure expert overflow rate, per-group discrepancy, routing entropy, validation perplexity, and loss variance across seeds. Success means fewer capacity overflows and lower training variance without worse perplexity, or improved perplexity at the same dropped-token rate.