Free-volume load balancing for sparse experts

Integrate this into a top-k mixture-of-experts router after computing softmax routing probabilities and before hard dispatch. For each batch, estimate q_i as the fraction of tokens sent to expert i, then smooth it with q_i ← (1 − γ)q_i + γ q_i_batch, where γ is an EMA coefficient. Normalize q_i so that its average equals the desired target occupancy. Set n_i = 1 − q_i and A_i = Δ_i + (1 − Δ_i)n_i. Add λ_FV R_FV to the router loss, with λ_FV and τ controlling the regularization strength. Pseudocode: logits = router(x); probabilities = softmax(logits); assignments = top_k(probabilities); q_batch = mean(assignments); q = (1 − γ)q + γ q_batch; n = 1 − q; A = Δ + (1 − Δ)n; loss = task_loss − λ_FV τ sum(log(A + ε)); backpropagate through the soft routing probabilities. The paper supplies the inverse-free-volume mechanism and asymptotic scaling; the occupancy estimator, Δ, and λ_FV must be selected empirically. First test a 16-expert top-1 MoE MLP on CIFAR-100 or WikiText-103 against no balancing and the standard Switch load-balancing loss. Sweep Δ from 0.001 to 0.1 and measure router-gradient magnitude versus n_i. The prediction is that gradient magnitude scales as 1/n_i for n_i much larger than Δ, then plateaus near 1/Δ for n_i much smaller than Δ. The crossover should occur near n_i = Δ/(1 − Δ), within approximately 20 percent.