Demographics and dynamics of Mechanical Turk workers

Djellel Difallah, Elena Filatova, Panos Ipeirotis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present an analysis of the population dynamics and demographics of Amazon Mechanical Turk workers based on the results of the survey that we conducted over a period of 28 months, with more than 85K responses from 40K unique participants. The demographics survey is ongoing (as of November 2017) and the results are available at http://demographics.mturk-tracker.com: we provide an API for researchers to download the survey data. We use techniques from the field of ecology, in particular, the capture-recapture technique, to understand the size and dynamics of the underlying population. We also demonstrate how to model and account for the inherent selection biases in such surveys. Our results indicate that there are more than 100K workers available in Amazon's crowdsourcing platform, the participation of the workers in the platform follows a heavy-tailed distribution, and at any given time there are more than 2K active workers. We also show that the half-life of a worker on the platform is around 12-18 months and that the rate of arrival of new workers balances the rate of departures, keeping the overall worker population relatively stable. Finally, we demonstrate how we can estimate the biases of different demographics to participate in the survey tasks, and show how to correct such biases. Our methodology is generic and can be applied to any platform where we are interested in understanding the dynamics and demographics of the underlying user population.

Original languageEnglish (US)
Title of host publicationWSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining
PublisherAssociation for Computing Machinery, Inc
Pages135-143
Number of pages9
ISBN (Electronic)9781450355810
DOIs
StatePublished - Feb 2 2018
Event11th ACM International Conference on Web Search and Data Mining, WSDM 2018 - Marina Del Rey, United States
Duration: Feb 5 2018Feb 9 2018

Publication series

NameWSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining
Volume2018-Febuary

Other

Other11th ACM International Conference on Web Search and Data Mining, WSDM 2018
CountryUnited States
CityMarina Del Rey
Period2/5/182/9/18

Keywords

  • Amazon mechanical Turk
  • Capture-recapture
  • Crowdsourcing
  • Demographics
  • Dynamics
  • Selection bias
  • Surveys

ASJC Scopus subject areas

  • Computer Science Applications
  • Software
  • Computer Networks and Communications
  • Information Systems

Fingerprint Dive into the research topics of 'Demographics and dynamics of Mechanical Turk workers'. Together they form a unique fingerprint.

  • Cite this

    Difallah, D., Filatova, E., & Ipeirotis, P. (2018). Demographics and dynamics of Mechanical Turk workers. In WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining (pp. 135-143). (WSDM 2018 - Proceedings of the 11th ACM International Conference on Web Search and Data Mining; Vol. 2018-Febuary). Association for Computing Machinery, Inc. https://doi.org/10.1145/3159652.3159661