Unverified 2026

Finite-domain survival-time MoE router

Usefulness5/10

Difficulty6/10

Novelty6/10

Dynamics Scalability Accuracy Stability Moe routing Training dynamics Transformer Stochastic processes Dynamical systems Probability Geometry

Source paper: Size-Dependent Growth Rates Amplify Infinitesimal Asymmetry in Nanocrystals arXiv:2609.00145 ⓘ · analyzed Sep 2, 2026

AI-generated research hypothesis, automatically tested. Not peer-reviewed.

Idea description

Replace a static top-k MoE capacity rule with a router whose expert allocation evolves through a finite-domain coverage process. Experts with larger current occupancy can either receive more future capacity, intentionally amplifying specialization, or receive less capacity by reversing the size dependence, allowing a controlled test of the paper's asymmetry-amplification mechanism.

Formulas

$$\tau_{\alpha}(\mathbf r;\mathcal D)=\int_{0}^{\infty}\exp[-N_{\alpha}(\mathbf r,t;\mathcal D)]\,dt$$

$$N_i(t;s_i)=\lambda_i(s_i)t^p,\qquad \lambda_i(s_i)=\lambda_0(s_i+\epsilon)^\beta$$

$$\tau_i(s_i)=\int_0^\infty e^{-\lambda_i(s_i)t^p}\,dt=\frac{\Gamma(1/p)}{p\,\lambda_i(s_i)^{1/p}},\qquad v_i(s_i)=\tau_i(s_i)^{-1}=\frac{p}{\Gamma(1/p)}\lambda_i(s_i)^{1/p}$$

$$s_i^{(u+1)}=\Pi_{[0,1]}\!\left(s_i^{(u)}+\eta\left[v_i(s_i^{(u)})-\frac1E\sum_{j=1}^{E}v_j(s_j^{(u)})\right]\right),\qquad \dot\delta_i\approx v'(\bar s)\delta_i$$

Mathematical statement

The paper defines the average survival time of a location $\mathbf r$ on facet class $\alpha$ and finite domain $\mathcal D$ as $\tau_\alpha(\mathbf r;\mathcal D)=\int_0^\infty \exp[-N_\alpha(\mathbf r,t;\mathcal D)]\,dt$. Here $N_\alpha(\mathbf r,t;\mathcal D)$ is the cumulative expected coverage or nucleation hazard by time $t$, and $\exp[-N_\alpha]$ is the probability that $\mathbf r$ remains uncovered. The transferable object is the finite-domain survival integral: boundary-limited coverage produces a characteristic completion time that depends on the instantaneous domain size. For an MoE router, expert $i$ has normalized occupancy $s_i\in[0,1]$, and we define a differentiable proxy hazard $N_i(t;s_i)=\lambda_i(s_i)t^p$, with $p>0$ and $\lambda_i(s_i)=\lambda_0(s_i+\epsilon)^\beta$. The resulting survival time is $\tau_i(s_i)=\int_0^\infty e^{-\lambda_i(s_i)t^p}dt=\Gamma(1/p)/(p\lambda_i(s_i)^{1/p})$. Set the expert's allocation-growth velocity to $v_i(s_i)=1/\tau_i(s_i)=c_p\lambda_i(s_i)^{1/p}$, where $c_p=p/\Gamma(1/p)$. For a continuous occupancy dynamics $\dot s_i=v_i(s_i)-\bar v$, a perturbation $\delta_i$ around equal occupancy obeys approximately $\dot\delta_i=v'(s)\delta_i$; therefore $\beta>0$ gives local asymmetry amplification and $\beta<0$ gives balancing feedback. The neural implementation should use a bounded discrete approximation of this dynamics rather than allowing unbounded expert capacity.

Implementation notes

Integrate the mechanism into the capacity/allocation state of a top-1 or top-2 MoE router, not into the expert MLP weights. Let the router produce logits $z_{b,i}$ for token $b$ and expert $i$, and let $q_{b,i}=\operatorname{softmax}(z_b)_i$. At each training step, compute soft occupancy $s_i=\operatorname{mean}_b q_{b,i}$, detach $s_i$ when updating the allocation state, and maintain a scalar capacity multiplier $c_i$. Initialize $c_i=1$. For every batch compute $q$, $s_i$, $\lambda_i=\lambda_0(s_i+\epsilon)^\beta$, and $v_i=p\lambda_i^{1/p}/\Gamma(1/p)$. Update $c_i\leftarrow\operatorname{clip}(c_i+\eta_c(v_i-\operatorname{mean}_jv_j),c_{\min},c_{\max})$, then renormalize so $\operatorname{mean}_i c_i=1$, and add $\log c_i$ to expert $i$'s router logit before top-k selection. Apply token dropping only after the modified logits. The paper-derived quantities are the survival integral, power-law hazard, inverse survival time, and centered velocity update; $\lambda_0,p,\beta,\eta_c,\epsilon$ are engineering hyperparameters. First run a 125M-parameter Switch-style Transformer on C4 or Wikitext-103 with 8 experts, comparing static capacity, standard load-balancing loss, $\beta=+1$ amplification, and $\beta=-1$ balancing at equal tokens, optimizer steps, expert FLOPs, and total parameter count. Measure validation perplexity, dropped-token fraction, expert utilization entropy, and the growth of an intentionally injected initial occupancy perturbation. Pre-register three mechanism tests: fit $\log|\delta_u|$ versus update number and compare its slope with $\eta_c v'(1/E)$ within 20%; verify that changing $\beta$ from $-1$ to $+1$ changes the slope from negative to positive with a transition near $\beta=0$; and verify that increasing $\eta_c$ increases the slope approximately linearly until clipping. Remove only the survival-time capacity update while retaining the same router and load-balancing loss. The proposal is falsified if occupancy perturbations do not show the predicted sign change, measured slopes are unrelated to $v'(1/E)$, or any perplexity/FLOP gain disappears when utilization and dropped tokens are matched.

Verification

This idea has not been verified yet.

Verification happens in two stages: Stage 1 — a mechanism check on a toy system confirms the claimed mathematical phenomenon reproduces; Stage 2 — a benchmark implements the idea on a real (small) neural network task and compares it against a tuned baseline over 8 paired seeds with a permutation test.

Artifacts

Artifacts unavailable.