Mutual-Invasibility Expert Router
Implementation & benchmark of arXiv:2607.11291 — Metacommunity persistence on spatially heterogeneous landscapes
Source paper: Metacommunity persistence on spatially heterogeneous landscapes arXiv:2607.11291 ⓘ · analyzed Aug 30, 2026
AI-generated research hypothesis, automatically tested. Not peer-reviewed.
Idea description
Construct a mixture-of-experts layer whose experts compete for a normalized routing resource, and regularize the router so that every expert can grow when introduced at low abundance into the equilibrium dominated by any other expert. The ecological mutual-invasibility criterion becomes a quantitative anti-collapse condition: if expert B has positive invasion growth against expert A's equilibrium and A has positive invasion growth against B, neither single-expert state is locally stable against replacement.
Formulas
Mathematical statement
For expert abundances p_i(t,x) greater than or equal to zero at environment or token state x, use the nonlocal competition dynamics dp_i/dt = -delta_i(x) p_i + (1 - sum_j p_j) integral K_i(x,y) p_i(y) dy. Here delta_i(x) is an effective expert extinction or dropout rate, K_i(x,y) is the routing-and-learning colonization kernel, and 1 - sum_j p_j(x) is remaining capacity. For two experts, let p_A_star be the equilibrium with p_B equal to zero and p_B_star the equilibrium with p_A equal to zero. Linearizing the rare-expert equation around p_A_star gives dq_B/dt = L_B_given_A q_B, where L_B_given_A q at x equals -delta_B(x) q(x) plus (1 - p_A_star(x)) times the integral of K_B(x,y) q(y) dy. Define the invasion rate lambda_B_given_A as the dominant eigenvalue or long-time growth rate of this positive operator. Mutual invasibility requires both lambda_B_given_A and lambda_A_given_B to be positive. In the neural implementation, p_i is an exponentially averaged routing occupancy, K_i is estimated from token transitions and loss-improvement statistics, and the positive eigenvalue is estimated by power iteration.
Implementation notes
Integrate this at the router level of a two- or four-expert MLP or Transformer MoE. For each minibatch, compute soft routing probabilities g_i(x), then maintain an occupancy field p_i(z) over a small set of token clusters z using p_i(z) = (1 - alpha) p_i(z) + alpha times the mean routing probability for expert i on cluster z. First train ordinary soft or sparse routing for a warm-up period. To estimate expert A's equilibrium, freeze the router, suppress expert B, and run several batches until p_A stabilizes. Estimate K_B(z,z') from a positive, exponentially smoothed transition matrix between source cluster z' and destination cluster z, weighted by B's routing probability and by inverse loss contribution. Use a small positive floor for numerical stability. Form the matrix L_B_given_A = -diag(delta_B) + (I - diag(p_A_star)) K_B and perform 10 to 20 power iterations, estimating the dominant growth rate from the logarithm of norm growth. Repeat with roles reversed every 200 to 1000 optimizer steps. Add beta times R_invasion to the loss, using stop-gradient through eigenvalue estimates initially. The paper supplies the nonlocal competition model and invasion criterion; K_i, delta_i, and clusters are empirical estimates. The first experiment is a two-expert CIFAR-10 MLP with deliberately imbalanced class clusters, compared with standard load balancing. The prediction is sharp: when both invasion rates cross zero, expert occupancy should stop collapsing; when one rate remains negative, the system should approach a single-expert state. Test whether the observed collapse boundary agrees with lambda equal to zero within 20 percent.
Verification
This idea has not been verified yet.
Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.
Artifacts
Artifacts unavailable.