arXiv:2608.08499 2026 Architecture 1 ideas extracted · analyzed Sep 1, 2026
The paper exposes a useful equivalence between a pairwise inner-product cut objective and vector balancing: maximizing cross-part interactions is equivalent, up to a constant, to minimizing the squared norms of aggregate feature vectors assigned to the parts. This yields a principled feature-aware alternative to scalar load balancing, with a simple greedy rule based on the incremental increase in aggregate norm. The most direct neural-network transfer is hard MoE routing: route tokens so that each expert receives a balanced aggregate of semantic feature vectors, rather than merely an equal number of tokens. The router is cheap and interpretable, and can be tested against capacity overflow, expert specialization, and quality at fixed communication cost.
Replace count-only MoE load balancing with greedy balancing of aggregate token-feature vectors. A token is assigned to the expert for which adding its feature vector produces the smallest increase in that expert's squared aggregate norm, encouraging experts to receive complementary semantic mixtures rather than identical token counts.