XNLI: Evaluating cross-lingual sentence representations

Alexis Conneau, Ruty Rinott, Guillaume Lample, Holger Schwenk, Ves Stoyanov, Adina Williams, Samuel R. Bowman

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a single language (usually English), and cannot be directly used beyond that language. Since collecting data in every language is not realistic, there has been a growing interest in crosslingual language understanding (XLU) and low-resource cross-language transfer. In this work, we construct an evaluation set for XLU by extending the development and test sets of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15 languages, including low-resource languages such as Swahili and Urdu. We hope that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence understanding by providing an informative standard evaluation task. In addition, we provide several baselines for multilingual sentence understanding, including two based on machine translation systems, and two that use parallel data to train aligned multilingual bag-of-words and LSTM encoders. We find that XNLI represents a practical and challenging evaluation suite, and that directly translating the test data yields the best performance among available baselines.

    Original languageEnglish (US)
    Title of host publicationProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
    EditorsEllen Riloff, David Chiang, Julia Hockenmaier, Jun'ichi Tsujii
    PublisherAssociation for Computational Linguistics
    Pages2475-2485
    Number of pages11
    ISBN (Electronic)9781948087841
    StatePublished - 2020
    Event2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 - Brussels, Belgium
    Duration: Oct 31 2018Nov 4 2018

    Publication series

    NameProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018

    Conference

    Conference2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
    CountryBelgium
    CityBrussels
    Period10/31/1811/4/18

    ASJC Scopus subject areas

    • Computational Theory and Mathematics
    • Computer Science Applications
    • Information Systems

    Fingerprint Dive into the research topics of 'XNLI: Evaluating cross-lingual sentence representations'. Together they form a unique fingerprint.

    Cite this