Vandermonde Expert Separation

Integrate the regularizer into the router of a top-1 or top-2 MoE Transformer. For each training batch, let r_b in R^d be the router input and let a_{be} be the pre-softmax router logit for expert e. Define the expert score s_e as the batch mean logit, s_e=B^{-1}\sum_b a_{be}; alternatively use a learned scalar score s_e=u^ op c_e, where c_e is an expert embedding or the mean hidden state of tokens routed to expert e. Standardize scores with z_e=(s_e-mean(s))/(std(s)+epsilon). Build V with V_{\ell e}=z_e^\ell for \ell=0,...,E-1. Compute the stabilized Gram matrix G=VV^ op+eta I and add -lambda_Vand*logdet(G) to the task loss. Use float32 for this calculation under mixed precision. Pseudocode: compute router logits a; compute ordinary routing probabilities and dispatch; compute s=mean(a, dim=batch); compute z=(s-mean(s))/(std(s)+eps); construct V=[z**ell for ell in range(E)]; set R=-slogdet(V@V.T+eta*I); set loss=task_loss+lambda_Vand*R+lambda_load*load_loss; backpropagate. The paper supplies the power-moment/Vandermonde construction and determinant criterion; score definitions, eta, and lambda are engineering choices. For E>16, use powers ell=0,...,q with q=4 to 8 rather than a full high-degree matrix. Start with a 12-layer, 8-expert Transformer on WikiText-103 or C4, preserving the baseline capacity factor, optimizer, and standard load-balancing loss. Measure validation perplexity, expert utilization entropy, dead-expert fraction, routing variance, and gradient norms at equal FLOPs. Success is fewer collapsed experts and lower perplexity at fixed active parameters; failure is no improvement over load balancing, increased routing oscillation, or exploding score magnitudes.