Glue: A multi-task benchmark and analysis platform for natural language understanding

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, Samuel R. Bowman

    Research output: Contribution to conferencePaper

    Abstract

    For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems.

    Original languageEnglish (US)
    StatePublished - 2019
    Event7th International Conference on Learning Representations, ICLR 2019 - New Orleans, United States
    Duration: May 6 2019May 9 2019

    Conference

    Conference7th International Conference on Learning Representations, ICLR 2019
    CountryUnited States
    CityNew Orleans
    Period5/6/195/9/19

    ASJC Scopus subject areas

    • Education
    • Computer Science Applications
    • Linguistics and Language
    • Language and Linguistics

    Fingerprint Dive into the research topics of 'Glue: A multi-task benchmark and analysis platform for natural language understanding'. Together they form a unique fingerprint.

  • Cite this

    Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). Glue: A multi-task benchmark and analysis platform for natural language understanding. Paper presented at 7th International Conference on Learning Representations, ICLR 2019, New Orleans, United States.