Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection

Alane Suhr, Clara Vania, Nikita Nangia, Maarten Sap, Mark Yatskar, Samuel R. Bowman, Yoav Artzi

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Crowdsourcing from non-experts is one of the most common approaches to collecting data and annotations in NLP. Even though it is such a fundamental tool in NLP, crowdsourcing use is largely guided by common practices and the personal experience of researchers. Developing a theory of crowdsourcing use for practical language problems remains an open challenge. However, there are various principles and practices that have proven effective in generating high quality and diverse data. This tutorial exposes NLP researchers to such data collection crowdsourcing methods and principles through a detailed discussion of a diverse set of case studies.

    Original languageEnglish (US)
    Title of host publicationEMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing
    Subtitle of host publicationTutorial Abstracts
    PublisherAssociation for Computational Linguistics (ACL)
    Pages1-6
    Number of pages6
    ISBN (Electronic)9781955917124
    StatePublished - 2021
    Event2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021 - Virtual, Punta Cana, Dominican Republic
    Duration: Nov 7 2021Nov 11 2021

    Publication series

    NameEMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts

    Conference

    Conference2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021
    Country/TerritoryDominican Republic
    CityVirtual, Punta Cana
    Period11/7/2111/11/21

    ASJC Scopus subject areas

    • Computational Theory and Mathematics
    • Computer Science Applications
    • Information Systems

    Fingerprint

    Dive into the research topics of 'Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection'. Together they form a unique fingerprint.

    Cite this