Annotation artifacts in natural language inference data

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, Noah A. Smith

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et al., 2015) and 53% of MultiNLI (Williams et al., 2018). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.

    Original languageEnglish (US)
    Title of host publicationShort Papers
    PublisherAssociation for Computational Linguistics (ACL)
    Pages107-112
    Number of pages6
    ISBN (Electronic)9781948087292
    StatePublished - 2018
    Event2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 - New Orleans, United States
    Duration: Jun 1 2018Jun 6 2018

    Publication series

    NameNAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
    Volume2

    Conference

    Conference2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018
    CountryUnited States
    CityNew Orleans
    Period6/1/186/6/18

    ASJC Scopus subject areas

    • Linguistics and Language
    • Language and Linguistics
    • Computer Science Applications

    Fingerprint Dive into the research topics of 'Annotation artifacts in natural language inference data'. Together they form a unique fingerprint.

  • Cite this

    Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S. R., & Smith, N. A. (2018). Annotation artifacts in natural language inference data. In Short Papers (pp. 107-112). (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 2). Association for Computational Linguistics (ACL).