TY - GEN
T1 - Demographics and dynamics of Mechanical Turk workers
AU - Difallah, Djellel
AU - Filatova, Elena
AU - Ipeirotis, Panos
N1 - Publisher Copyright:
© 2018 Association for Computing Machinery.
PY - 2018/2/2
Y1 - 2018/2/2
N2 - We present an analysis of the population dynamics and demographics of Amazon Mechanical Turk workers based on the results of the survey that we conducted over a period of 28 months, with more than 85K responses from 40K unique participants. The demographics survey is ongoing (as of November 2017) and the results are available at http://demographics.mturk-tracker.com: we provide an API for researchers to download the survey data. We use techniques from the field of ecology, in particular, the capture-recapture technique, to understand the size and dynamics of the underlying population. We also demonstrate how to model and account for the inherent selection biases in such surveys. Our results indicate that there are more than 100K workers available in Amazon's crowdsourcing platform, the participation of the workers in the platform follows a heavy-tailed distribution, and at any given time there are more than 2K active workers. We also show that the half-life of a worker on the platform is around 12-18 months and that the rate of arrival of new workers balances the rate of departures, keeping the overall worker population relatively stable. Finally, we demonstrate how we can estimate the biases of different demographics to participate in the survey tasks, and show how to correct such biases. Our methodology is generic and can be applied to any platform where we are interested in understanding the dynamics and demographics of the underlying user population.
AB - We present an analysis of the population dynamics and demographics of Amazon Mechanical Turk workers based on the results of the survey that we conducted over a period of 28 months, with more than 85K responses from 40K unique participants. The demographics survey is ongoing (as of November 2017) and the results are available at http://demographics.mturk-tracker.com: we provide an API for researchers to download the survey data. We use techniques from the field of ecology, in particular, the capture-recapture technique, to understand the size and dynamics of the underlying population. We also demonstrate how to model and account for the inherent selection biases in such surveys. Our results indicate that there are more than 100K workers available in Amazon's crowdsourcing platform, the participation of the workers in the platform follows a heavy-tailed distribution, and at any given time there are more than 2K active workers. We also show that the half-life of a worker on the platform is around 12-18 months and that the rate of arrival of new workers balances the rate of departures, keeping the overall worker population relatively stable. Finally, we demonstrate how we can estimate the biases of different demographics to participate in the survey tasks, and show how to correct such biases. Our methodology is generic and can be applied to any platform where we are interested in understanding the dynamics and demographics of the underlying user population.
KW - Amazon mechanical Turk
KW - Capture-recapture
KW - Crowdsourcing
KW - Demographics
KW - Dynamics
KW - Selection bias
KW - Surveys
UR - http://www.scopus.com/inward/record.url?scp=85046905206&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85046905206&partnerID=8YFLogxK
U2 - 10.1145/3159652.3159661
DO - 10.1145/3159652.3159661
M3 - Conference contribution
AN - SCOPUS:85046905206
T3 - WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining
SP - 135
EP - 143
BT - WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining
PB - Association for Computing Machinery, Inc
T2 - 11th ACM International Conference on Web Search and Data Mining, WSDM 2018
Y2 - 5 February 2018 through 9 February 2018
ER -