A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory

Jon Sprouse

Research output: Contribution to journalArticlepeer-review

Abstract

Amazon's Mechanical Turk (AMT) is a Web application that provides instant access to thousands of potential participants for survey-based psychology experiments, such as the acceptability judgment task used extensively in syntactic theory. Because AMT is a Web-based system, syntacticians may worry that the move out of the experimenter-controlled environment of the laboratory and onto the user-controlled environment of AMT could adversely affect the quality of the judgment data collected. This article reports a quantitative comparison of two identical acceptability judgment experiments, each with 176 participants (352 total): one conducted in the laboratory, and one conducted on AMT. Crucial indicators of data quality-such as participant rejection rates, statistical power, and the shape of the distributions of the judgments for each sentence type-are compared between the two samples. The results suggest that aside from slightly higher participant rejection rates, AMT data are almost indistinguishable from laboratory data.

Original languageEnglish (US)
Pages (from-to)155-167
Number of pages13
JournalBehavior Research Methods
Volume43
Issue number1
DOIs
StatePublished - Mar 2011

Keywords

  • Acceptability judgments
  • Amazon Mechanical Turk
  • Experimental syntax
  • Grammaticality judgments
  • Linguistic theory

ASJC Scopus subject areas

  • Experimental and Cognitive Psychology
  • Developmental and Educational Psychology
  • Arts and Humanities (miscellaneous)
  • Psychology (miscellaneous)
  • Psychology(all)

Fingerprint

Dive into the research topics of 'A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory'. Together they form a unique fingerprint.

Cite this