Colorless green recurrent networks dream hierarchically

Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, Marco Baroni

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Recurrent neural networks (RNNs) have achieved impressive results in a variety of linguistic processing tasks, suggesting that they can induce non-Trivial properties of language. We investigate here to what extent RNNs learn to track abstract hierarchical syntactic structure. We test whether RNNs trained with a generic language modeling objective in four languages (Italian, English, Hebrew, Russian) can predict long-distance number agreement in various constructions. We include in our evaluation nonsensical sentences where RNNs cannot rely on semantic or lexical cues ("The colorless green iiiiiiiiiiiiiiddddddddddddeeeeeeeeeeeeeeaaaaaaaaaaaaaassssssssssssss I ate with the chair sssssssssssssslllllllllllllleeeeeeeeeeeeeeeeeeeeeeeeeeeepppppppppppp furiously"), and, for Italian, we compare model performance to human intuitions. Our language-model-Trained RNNs make reliable predictions about long-distance agreement, and do not lag much behind human performance. We thus bring support to the hypothesis that RNNs are not just shallowpattern extractors, but they also acquire deeper grammatical competence.

    Original languageEnglish (US)
    Title of host publicationLong Papers
    PublisherAssociation for Computational Linguistics (ACL)
    Pages1195-1205
    Number of pages11
    ISBN (Electronic)9781948087278
    StatePublished - 2018
    Event2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018 - New Orleans, United States
    Duration: Jun 1 2018Jun 6 2018

    Publication series

    NameNAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
    Volume1

    Conference

    Conference2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018
    CountryUnited States
    CityNew Orleans
    Period6/1/186/6/18

    ASJC Scopus subject areas

    • Linguistics and Language
    • Language and Linguistics
    • Computer Science Applications

    Fingerprint Dive into the research topics of 'Colorless green recurrent networks dream hierarchically'. Together they form a unique fingerprint.

    Cite this