Exploring the syntactic abilities of RNNs with multi-task learning

Émile Enguehard, Yoav Goldberg, Tal Linzen

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Recent work has explored the syntactic abilities of RNNs using the subject-verb agreement task, which diagnoses sensitivity to sentence structure. RNNs performed this task well in common cases, but faltered in complex sentences (Linzen et al., 2016). We test whether these errors are due to inherent limitations of the architecture or to the relatively indirect supervision provided by most agreement dependencies in a corpus. We trained a single RNN to perform both the agreement task and an additional task, either CCG supertagging or language modeling. Multitask training led to significantly lower error rates, in particular on complex sentences, suggesting that RNNs have the ability to evolve more sophisticated syntactic representations than shown before. We also show that easily available agreement training data can improve performance on other syntactic tasks, in particular when only a limited amount of training data is available for those tasks. The multi-task paradigm can also be leveraged to inject grammatical knowledge into language models.

    Original languageEnglish (US)
    Title of host publicationCoNLL 2017 - 21st Conference on Computational Natural Language Learning, Proceedings
    PublisherAssociation for Computational Linguistics (ACL)
    Pages3-14
    Number of pages12
    ISBN (Electronic)9781945626548
    DOIs
    StatePublished - 2017
    Event21st Conference on Computational Natural Language Learning, CoNLL 2017 - Vancouver, Canada
    Duration: Aug 3 2017Aug 4 2017

    Publication series

    NameCoNLL 2017 - 21st Conference on Computational Natural Language Learning, Proceedings

    Conference

    Conference21st Conference on Computational Natural Language Learning, CoNLL 2017
    Country/TerritoryCanada
    CityVancouver
    Period8/3/178/4/17

    ASJC Scopus subject areas

    • Linguistics and Language
    • Artificial Intelligence
    • Human-Computer Interaction

    Fingerprint

    Dive into the research topics of 'Exploring the syntactic abilities of RNNs with multi-task learning'. Together they form a unique fingerprint.

    Cite this