TY - JOUR
T1 - Neural Network Acceptability Judgments
AU - Warstadt, Alex
AU - Singh, Amanpreet
AU - Bowman, Samuel R.
N1 - Funding Information:
This project has benefited from help and feedback at various stages from Chris Barker, Pablo Gonzalez, Shalom Lappin, Omer Levy, Marie-Catherine de Marneffe, Alex Wang, Alexander Clark, everyone in the Deep Learning in Semantics seminar at NYU, and three anonymous TACL reviewers. This project has benefited from financial support to S.B. by Google, Tencent Holdings, and Samsung Research. This material is based upon work supported by the National Science Foundation under grant no. 1850208. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Publisher Copyright:
© 2019 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license.
PY - 2019
Y1 - 2019
N2 - This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verbobject order. However, all models we test perform far below human level on a wide range of grammatical constructions.
AB - This paper investigates the ability of artificial neural networks to judge the grammatical acceptability of a sentence, with the goal of testing their linguistic competence. We introduce the Corpus of Linguistic Acceptability (CoLA), a set of 10,657 English sentences labeled as grammatical or ungrammatical from published linguistics literature. As baselines, we train several recurrent neural network models on acceptability classification, and find that our models outperform unsupervised models by Lau et al. (2016) on CoLA. Error-analysis on specific grammatical phenomena reveals that both Lau et al.’s models and ours learn systematic generalizations like subject-verbobject order. However, all models we test perform far below human level on a wide range of grammatical constructions.
UR - http://www.scopus.com/inward/record.url?scp=85077358198&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85077358198&partnerID=8YFLogxK
U2 - 10.1162/tacl_a_00290
DO - 10.1162/tacl_a_00290
M3 - Article
AN - SCOPUS:85077358198
SN - 2307-387X
VL - 7
SP - 625
EP - 641
JO - Transactions of the Association for Computational Linguistics
JF - Transactions of the Association for Computational Linguistics
ER -