Annotation artifacts in natural language inference data

Suchin Gururangan; Swabha Swayamdipta; Omer Levy; Roy Schwartz; Samuel R. Bowman; Noah A. Smith

Annotation artifacts in natural language inference data

Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, Noah A. Smith

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et al., 2015) and 53% of MultiNLI (Williams et al., 2018). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.

Original language	English (US)
Title of host publication	Short Papers
Publisher	Association for Computational Linguistics (ACL)
Pages	107-112
Number of pages	6
ISBN (Electronic)	9781948087292
State	Published - 2018
Event	2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 - New Orleans, United States Duration: Jun 1 2018 → Jun 6 2018

Publication series

Name	NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Volume	2

Conference

Conference	2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018
Country/Territory	United States
City	New Orleans
Period	6/1/18 → 6/6/18

ASJC Scopus subject areas

Linguistics and Language
Language and Linguistics
Computer Science Applications

Cite this

Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S. R., & Smith, N. A. (2018). Annotation artifacts in natural language inference data. In Short Papers (pp. 107-112). (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 2). Association for Computational Linguistics (ACL).

Annotation artifacts in natural language inference data. / Gururangan, Suchin; Swayamdipta, Swabha; Levy, Omer et al.
Short Papers. Association for Computational Linguistics (ACL), 2018. p. 107-112 (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference; Vol. 2).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Gururangan, S, Swayamdipta, S, Levy, O, Schwartz, R, Bowman, SR & Smith, NA 2018, Annotation artifacts in natural language inference data. in Short Papers. NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, vol. 2, Association for Computational Linguistics (ACL), pp. 107-112, 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018, New Orleans, United States, 6/1/18.

Gururangan S, Swayamdipta S, Levy O, Schwartz R, Bowman SR, Smith NA. Annotation artifacts in natural language inference data. In Short Papers. Association for Computational Linguistics (ACL). 2018. p. 107-112. (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference).

Gururangan, Suchin ; Swayamdipta, Swabha ; Levy, Omer et al. / Annotation artifacts in natural language inference data. Short Papers. Association for Computational Linguistics (ACL), 2018. pp. 107-112 (NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference).

@inproceedings{7a033d38d8b1407c960b695aaa326d8c,

title = "Annotation artifacts in natural language inference data",

abstract = "Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et al., 2015) and 53% of MultiNLI (Williams et al., 2018). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.",

author = "Suchin Gururangan and Swabha Swayamdipta and Omer Levy and Roy Schwartz and Bowman, {Samuel R.} and Smith, {Noah A.}",

note = "Publisher Copyright: {\textcopyright} 2018 Association for Computational Linguistics.; 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 ; Conference date: 01-06-2018 Through 06-06-2018",

year = "2018",

language = "English (US)",

series = "NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference",

publisher = "Association for Computational Linguistics (ACL)",

pages = "107--112",

booktitle = "Short Papers",

}

TY - GEN

T1 - Annotation artifacts in natural language inference data

AU - Gururangan, Suchin

AU - Swayamdipta, Swabha

AU - Levy, Omer

AU - Schwartz, Roy

AU - Bowman, Samuel R.

AU - Smith, Noah A.

PY - 2018

Y1 - 2018

N2 - Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et al., 2015) and 53% of MultiNLI (Williams et al., 2018). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.

AB - Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to. We show that, in a significant portion of such data, this protocol leaves clues that make it possible to identify the label by looking only at the hypothesis, without observing the premise. Specifically, we show that a simple text categorization model can correctly classify the hypothesis alone in about 67% of SNLI (Bowman et al., 2015) and 53% of MultiNLI (Williams et al., 2018). Our analysis reveals that specific linguistic phenomena such as negation and vagueness are highly correlated with certain inference classes. Our findings suggest that the success of natural language inference models to date has been overestimated, and that the task remains a hard open problem.

UR - http://www.scopus.com/inward/record.url?scp=85063087625&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063087625&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85063087625

T3 - NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference

SP - 107

EP - 112

BT - Short Papers

PB - Association for Computational Linguistics (ACL)

T2 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018

Y2 - 1 June 2018 through 6 June 2018

ER -

Annotation artifacts in natural language inference data

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this