The reliability of acceptability judgments across languages

Tal Linzen, Yohei Oseki

    Research output: Contribution to journalReview articlepeer-review


    The reliability of acceptability judgments made by individual linguists has often been called into question. Recent large-scale replication studies conducted in response to this criticism have shown that the majority of published English acceptability judgments are robust. We make two observations about these replication studies. First, we raise the concern that English acceptability judgments may be more reliable than judgments in other languages. Second, we argue that it is unnecessary to replicate judgments that illustrate uncontroversial descriptive facts; rather, candidates for replication can emerge during formal or informal peer review. We present two experiments motivated by these arguments. Published Hebrew and Japanese acceptability contrasts considered questionable by the authors of the present paper were rated for acceptability by a large sample of naive participants. Approximately half of the contrasts did not replicate. We suggest that the reliability of acceptability judgments, especially in languages other than English, can be improved using a simple open review system, and that formal experiments are only necessary in controversial cases.

    Original languageEnglish (US)
    Article number100
    Issue number1
    StatePublished - 2019


    • Acceptability judgments
    • Experimental syntax
    • Hebrew
    • Japanese
    • Reliability

    ASJC Scopus subject areas

    • Linguistics and Language
    • Language and Linguistics


    Dive into the research topics of 'The reliability of acceptability judgments across languages'. Together they form a unique fingerprint.

    Cite this