TY - GEN
T1 - What Do NLP Researchers Believe? Results of the NLP Community Metasurvey
AU - Michael, Julian
AU - Holtzman, Ari
AU - Parrish, Alicia
AU - Mueller, Aaron
AU - Wang, Alex
AU - Chen, Angelica
AU - Madaan, Divyam
AU - Nangia, Nikita
AU - Pang, Richard Yuanzhe
AU - Phang, Jason
AU - Bowman, Samuel R.
N1 - Publisher Copyright:
© 2023 Association for Computational Linguistics.
PY - 2023
Y1 - 2023
N2 - We present the results of the NLP Community Metasurvey. Run from May to June 2022, it elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split in half on the importance of artificial general intelligence, whether language models understand language, and the necessity of linguistic structure and inductive bias for solving NLP problems. In addition, the survey posed meta-questions, asking respondents to predict the distribution of survey responses. This allows us to uncover false sociological beliefs where the community's predictions don't match reality. Among other results, we find that the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.
AB - We present the results of the NLP Community Metasurvey. Run from May to June 2022, it elicited opinions on controversial issues, including industry influence in the field, concerns about AGI, and ethics. Our results put concrete numbers to several controversies: For example, respondents are split in half on the importance of artificial general intelligence, whether language models understand language, and the necessity of linguistic structure and inductive bias for solving NLP problems. In addition, the survey posed meta-questions, asking respondents to predict the distribution of survey responses. This allows us to uncover false sociological beliefs where the community's predictions don't match reality. Among other results, we find that the community greatly overestimates its own belief in the usefulness of benchmarks and the potential for scaling to solve real-world problems, while underestimating its belief in the importance of linguistic structure, inductive bias, and interdisciplinary science.
UR - http://www.scopus.com/inward/record.url?scp=85174425838&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85174425838&partnerID=8YFLogxK
U2 - 10.18653/v1/2023.acl-long.903
DO - 10.18653/v1/2023.acl-long.903
M3 - Conference contribution
AN - SCOPUS:85174425838
T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics
SP - 16334
EP - 16368
BT - Long Papers
PB - Association for Computational Linguistics (ACL)
T2 - 61st Annual Meeting of the Association for Computational Linguistics, ACL 2023
Y2 - 9 July 2023 through 14 July 2023
ER -