Cross-linguistic syntactic evaluation of word prediction models

Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, Tal Linzen

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    A range of studies have concluded that neural word prediction models can distinguish grammatical from ungrammatical sentences with high accuracy. However, these studies are based primarily on monolingual evidence from English. To investigate how these models' ability to learn syntax varies by language, we introduce CLAMS (Cross-Linguistic Assessment of Models on Syntax), a syntactic evaluation suite for monolingual and multilingual models. CLAMS includes subject-verb agreement challenge sets for English, French, German, Hebrew and Russian, generated from grammars we develop. We use CLAMS to evaluate LSTM language models as well as monolingual and multilingual BERT. Across languages, monolingual LSTMs achieved high accuracy on dependencies without attractors, and generally poor accuracy on agreement across object relative clauses. On other constructions, agreement accuracy was generally higher in languages with richer morphology. Multilingual models generally underperformed monolingual models. Multilingual BERT showed high syntactic accuracy on English, but noticeable deficiencies in other languages.

    Original languageEnglish (US)
    Title of host publicationACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
    PublisherAssociation for Computational Linguistics (ACL)
    Number of pages17
    ISBN (Electronic)9781952148255
    StatePublished - 2020
    Event58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 - Virtual, Online, United States
    Duration: Jul 5 2020Jul 10 2020

    Publication series

    NameProceedings of the Annual Meeting of the Association for Computational Linguistics
    ISSN (Print)0736-587X


    Conference58th Annual Meeting of the Association for Computational Linguistics, ACL 2020
    Country/TerritoryUnited States
    CityVirtual, Online

    ASJC Scopus subject areas

    • Computer Science Applications
    • Linguistics and Language
    • Language and Linguistics


    Dive into the research topics of 'Cross-linguistic syntactic evaluation of word prediction models'. Together they form a unique fingerprint.

    Cite this