How can we accelerate progress towards human-like linguistic generalization?

Tal Linzen

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has become a central tool for measuring progress in natural language understanding. This paradigm consists of three stages: (1) pretraining of a word prediction model on a corpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set representing a classification task; (3) evaluation on a test set drawn from the same distribution as that training set. This paradigm favors simple, low-bias architectures, which, first, can be scaled to process vast amounts of data, and second, can capture the fine-grained statistical properties of a particular data set, regardless of whether those properties are likely to generalize to examples of the task outside the data set. This contrasts with humans, who learn language from several orders of magnitude less data than the systems favored by this evaluation paradigm, and generalize to new tasks in a consistent way. We advocate for supplementing or replacing PAID with paradigms that reward architectures that generalize as quickly and robustly as humans.

    Original languageEnglish (US)
    Title of host publicationACL 2020 - 58th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference
    PublisherAssociation for Computational Linguistics (ACL)
    Pages5210-5217
    Number of pages8
    ISBN (Electronic)9781952148255
    DOIs
    StatePublished - 2020
    Event58th Annual Meeting of the Association for Computational Linguistics, ACL 2020 - Virtual, Online, United States
    Duration: Jul 5 2020Jul 10 2020

    Publication series

    NameProceedings of the Annual Meeting of the Association for Computational Linguistics
    ISSN (Print)0736-587X

    Conference

    Conference58th Annual Meeting of the Association for Computational Linguistics, ACL 2020
    Country/TerritoryUnited States
    CityVirtual, Online
    Period7/5/207/10/20

    ASJC Scopus subject areas

    • Computer Science Applications
    • Linguistics and Language
    • Language and Linguistics

    Fingerprint

    Dive into the research topics of 'How can we accelerate progress towards human-like linguistic generalization?'. Together they form a unique fingerprint.

    Cite this