What Do NLP Researchers Believe? Results of the NLP Community Metasurvey

Julian Michael, Ari Holtzman, Alicia Parrish, Aaron Mueller, Alex Wang, Angelica Chen, Divyam Madaan, Nikita Nangia, Richard Yuanzhe Pang, Jason Phang, Samuel R. Bowman

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    We present the results of the NLP Community Metasurvey. Run from May to June 2022, it elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split in half on the importance of artificial general intelligence, whether language models understand language, and the necessity of linguistic structure and inductive bias for solving NLP problems. In addition, the survey posed meta-questions, asking respondents to predict the distribution of survey responses. This allows us to uncover false sociological beliefs where the community's predictions don't match reality. Among other results, we find that the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.

    Original languageEnglish (US)
    Title of host publicationLong Papers
    PublisherAssociation for Computational Linguistics (ACL)
    Number of pages35
    ISBN (Electronic)9781959429722
    StatePublished - 2023
    Event61st Annual Meeting of the Association for Computational Linguistics, ACL 2023 - Toronto, Canada
    Duration: Jul 9 2023Jul 14 2023

    Publication series

    NameProceedings of the Annual Meeting of the Association for Computational Linguistics
    ISSN (Print)0736-587X


    Conference61st Annual Meeting of the Association for Computational Linguistics, ACL 2023

    ASJC Scopus subject areas

    • Computer Science Applications
    • Linguistics and Language
    • Language and Linguistics

    Cite this