Assessing the reliability of textbook data in syntax: Adger's Core Syntax

Jon Sprouse, Diogo Almeida

Research output: Contribution to journalArticlepeer-review

Abstract

There has been a consistent pattern of criticism of the reliability of acceptability judgment data in syntax for at least 50 years (e.g., Hill 1961), culminating in several high-profile criticisms within the past ten years (Edelman & Christiansen 2003, Ferreira 2005, Wasow & Arnold 2005, Gibson & Fedorenko 2010, in press). The fundamental claim of these critics is that traditional acceptability judgment collection methods, which tend to be relatively informal compared to methods from experimental psychology, lead to an intolerably high number of false positive results. In this paper we empirically assess this claim by formally testing all 469 (unique, US-English) data points from a popular syntax textbook (Adger 2003) using 440 naïve participants, two judgment tasks (magnitude estimation and yes-no), and three different types of statistical analyses (standard frequentist tests, linear mixed effects models, and Bayes factor analyses). The results suggest that the maximum discrepancy between traditional methods and formal experimental methods is 2%. This suggests that even under the (likely unwarranted) assumption that the discrepant results are all false positives that have found their way into the syntactic literature due to the shortcomings of traditional methods, the minimum replication rate of these 469 data points is 98%. We discuss the implications of these results for questions about the reliability of syntactic data, as well as the practical consequences of these results for the methodological options available to syntacticians.

Original languageEnglish (US)
Pages (from-to)609-652
Number of pages44
JournalJournal of Linguistics
Volume48
Issue number3
DOIs
StatePublished - Oct 2012

ASJC Scopus subject areas

  • Language and Linguistics
  • Philosophy
  • Linguistics and Language

Fingerprint

Dive into the research topics of 'Assessing the reliability of textbook data in syntax: Adger's Core Syntax'. Together they form a unique fingerprint.

Cite this