longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks

Venelin Kovatchev; Trina Chatterjee; Venkata S. Govindarajan; Jifan Chen; Eunsol Choi; Gabriella Chronis; Anubrata Das; Katrin Erk; Matthew Lease; Junyi Jessy Li; Yating Wu; Kyle Mahowald

longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks

Venelin Kovatchev, Trina Chatterjee, Venkata S. Govindarajan, Jifan Chen, Eunsol Choi, Gabriella Chronis, Anubrata Das, Katrin Erk, Matthew Lease, Junyi Jessy Li, Yating Wu, Kyle Mahowald

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team “longhorns” on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first, with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.

Original language	English (US)
Title of host publication	DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop
Editors	Max Bartolo, Hannah Rose Kirk, Pedro Rodriguez, Katerina Margatina, Tristan Thrush, Robin Jia, Pontus Stenetorp, Adina Williams, Douwe Kiela
Publisher	Association for Computational Linguistics (ACL)
Pages	41-52
Number of pages	12
ISBN (Electronic)	9781955917940
State	Published - 2022
Event	1st Workshop on Dynamic Adversarial Data Collection, DADC 2022 - Seattle, United States Duration: Jul 14 2022 → …

Publication series

Name	DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop

Conference

Conference	1st Workshop on Dynamic Adversarial Data Collection, DADC 2022
Country/Territory	United States
City	Seattle
Period	7/14/22 → …

ASJC Scopus subject areas

Language and Linguistics
Computer Science Applications
Linguistics and Language

Cite this

Kovatchev, V., Chatterjee, T., Govindarajan, V. S., Chen, J., Choi, E., Chronis, G., Das, A., Erk, K., Lease, M., Li, J. J., Wu, Y., & Mahowald, K. (2022). longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. In M. Bartolo, H. R. Kirk, P. Rodriguez, K. Margatina, T. Thrush, R. Jia, P. Stenetorp, A. Williams, & D. Kiela (Eds.), DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop (pp. 41-52). (DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop). Association for Computational Linguistics (ACL).

longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. / Kovatchev, Venelin; Chatterjee, Trina; Govindarajan, Venkata S. et al.
DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop. ed. / Max Bartolo; Hannah Rose Kirk; Pedro Rodriguez; Katerina Margatina; Tristan Thrush; Robin Jia; Pontus Stenetorp; Adina Williams; Douwe Kiela. Association for Computational Linguistics (ACL), 2022. p. 41-52 (DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kovatchev, V, Chatterjee, T, Govindarajan, VS, Chen, J, Choi, E, Chronis, G, Das, A, Erk, K, Lease, M, Li, JJ, Wu, Y & Mahowald, K 2022, longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. in M Bartolo, HR Kirk, P Rodriguez, K Margatina, T Thrush, R Jia, P Stenetorp, A Williams & D Kiela (eds), DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop. DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop, Association for Computational Linguistics (ACL), pp. 41-52, 1st Workshop on Dynamic Adversarial Data Collection, DADC 2022, Seattle, United States, 7/14/22.

Kovatchev V, Chatterjee T, Govindarajan VS, Chen J, Choi E, Chronis G et al. longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. In Bartolo M, Kirk HR, Rodriguez P, Margatina K, Thrush T, Jia R, Stenetorp P, Williams A, Kiela D, editors, DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop. Association for Computational Linguistics (ACL). 2022. p. 41-52. (DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop).

Kovatchev, Venelin ; Chatterjee, Trina ; Govindarajan, Venkata S. et al. / longhorns at DADC 2022 : How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks. DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop. editor / Max Bartolo ; Hannah Rose Kirk ; Pedro Rodriguez ; Katerina Margatina ; Tristan Thrush ; Robin Jia ; Pontus Stenetorp ; Adina Williams ; Douwe Kiela. Association for Computational Linguistics (ACL), 2022. pp. 41-52 (DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop).

@inproceedings{6e7cf23296784e0d8437c0a2899b4c6d,

title = "longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks",

abstract = "Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team “longhorns” on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first, with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.",

author = "Venelin Kovatchev and Trina Chatterjee and Govindarajan, {Venkata S.} and Jifan Chen and Eunsol Choi and Gabriella Chronis and Anubrata Das and Katrin Erk and Matthew Lease and Li, {Junyi Jessy} and Yating Wu and Kyle Mahowald",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 1st Workshop on Dynamic Adversarial Data Collection, DADC 2022 ; Conference date: 14-07-2022",

year = "2022",

language = "English (US)",

series = "DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop",

publisher = "Association for Computational Linguistics (ACL)",

pages = "41--52",

editor = "Max Bartolo and Kirk, {Hannah Rose} and Pedro Rodriguez and Katerina Margatina and Tristan Thrush and Robin Jia and Pontus Stenetorp and Adina Williams and Douwe Kiela",

booktitle = "DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop",

}

TY - GEN

T1 - longhorns at DADC 2022

T2 - 1st Workshop on Dynamic Adversarial Data Collection, DADC 2022

AU - Kovatchev, Venelin

AU - Chatterjee, Trina

AU - Govindarajan, Venkata S.

AU - Chen, Jifan

AU - Choi, Eunsol

AU - Chronis, Gabriella

AU - Das, Anubrata

AU - Erk, Katrin

AU - Lease, Matthew

AU - Li, Junyi Jessy

AU - Wu, Yating

AU - Mahowald, Kyle

PY - 2022

Y1 - 2022

N2 - Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team “longhorns” on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first, with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.

AB - Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability. Here, we describe the approach of the team “longhorns” on Task 1 of the The First Workshop on Dynamic Adversarial Data Collection (DADC), which asked teams to manually fool a model on an Extractive Question Answering task. Our team finished first, with a model error rate of 62%. We advocate for a systematic, linguistically informed approach to formulating adversarial questions, and we describe the results of our pilot experiments, as well as our official submission.

UR - http://www.scopus.com/inward/record.url?scp=85137596084&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85137596084&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85137596084

T3 - DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop

SP - 41

EP - 52

BT - DADC 2022 - 1st Workshop on Dynamic Adversarial Data Collection, Proceedings of the Workshop

A2 - Bartolo, Max

A2 - Kirk, Hannah Rose

A2 - Rodriguez, Pedro

A2 - Margatina, Katerina

A2 - Thrush, Tristan

A2 - Jia, Robin

A2 - Stenetorp, Pontus

A2 - Williams, Adina

A2 - Kiela, Douwe

PB - Association for Computational Linguistics (ACL)

Y2 - 14 July 2022

ER -

longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this