QUAC: Question answering in context

Eunsol Choi; He He; Mohit Iyyer; Mark Yatskar; Wen Tau Yih; Yejin Choi; Percy Liang; Luke Zettlemoyer

QUAC: Question answering in context

Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen Tau Yih, Yejin Choi, Percy Liang, Luke Zettlemoyer

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation. We also report results for a number of reference models, including a recently state-of-the-art reading comprehension architecture extended to model dialog context. Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data. Dataset, baseline, and leaderboard available at http://quac.ai.

Original language	English (US)
Title of host publication	Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
Editors	Ellen Riloff, David Chiang, Julia Hockenmaier, Jun'ichi Tsujii
Publisher	Association for Computational Linguistics
Pages	2174-2184
Number of pages	11
ISBN (Electronic)	9781948087841
State	Published - Jan 1 2020
Event	2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 - Brussels, Belgium Duration: Oct 31 2018 → Nov 4 2018

Publication series

Name	Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018

Conference

Conference	2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
Country/Territory	Belgium
City	Brussels
Period	10/31/18 → 11/4/18

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Science Applications
Information Systems

Cite this

Choi, E., He, H., Iyyer, M., Yatskar, M., Yih, W. T., Choi, Y., Liang, P., & Zettlemoyer, L. (2020). QUAC: Question answering in context. In E. Riloff, D. Chiang, J. Hockenmaier, & J. Tsujii (Eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 (pp. 2174-2184). (Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018). Association for Computational Linguistics.

QUAC: Question answering in context. / Choi, Eunsol; He, He; Iyyer, Mohit et al.
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. ed. / Ellen Riloff; David Chiang; Julia Hockenmaier; Jun'ichi Tsujii. Association for Computational Linguistics, 2020. p. 2174-2184 (Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Choi, E, He, H, Iyyer, M, Yatskar, M, Yih, WT, Choi, Y, Liang, P & Zettlemoyer, L 2020, QUAC: Question answering in context. in E Riloff, D Chiang, J Hockenmaier & J Tsujii (eds), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, Association for Computational Linguistics, pp. 2174-2184, 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, Brussels, Belgium, 10/31/18.

Choi E, He H, Iyyer M, Yatskar M, Yih WT, Choi Y et al. QUAC: Question answering in context. In Riloff E, Chiang D, Hockenmaier J, Tsujii J, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. Association for Computational Linguistics. 2020. p. 2174-2184. (Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018).

Choi, Eunsol ; He, He ; Iyyer, Mohit et al. / QUAC : Question answering in context. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018. editor / Ellen Riloff ; David Chiang ; Julia Hockenmaier ; Jun'ichi Tsujii. Association for Computational Linguistics, 2020. pp. 2174-2184 (Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018).

@inproceedings{aabf6709a4944ac8abfc1a4c8286c0ed,

title = "QUAC: Question answering in context",

abstract = "We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation. We also report results for a number of reference models, including a recently state-of-the-art reading comprehension architecture extended to model dialog context. Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data. Dataset, baseline, and leaderboard available at http://quac.ai.",

author = "Eunsol Choi and He He and Mohit Iyyer and Mark Yatskar and Yih, {Wen Tau} and Yejin Choi and Percy Liang and Luke Zettlemoyer",

year = "2020",

month = jan,

day = "1",

language = "English (US)",

series = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018",

publisher = "Association for Computational Linguistics",

pages = "2174--2184",

editor = "Ellen Riloff and David Chiang and Julia Hockenmaier and Jun'ichi Tsujii",

booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018",

note = "2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 ; Conference date: 31-10-2018 Through 04-11-2018",

}

TY - GEN

T1 - QUAC

T2 - 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018

AU - Choi, Eunsol

AU - He, He

AU - Iyyer, Mohit

AU - Yatskar, Mark

AU - Yih, Wen Tau

AU - Choi, Yejin

AU - Liang, Percy

AU - Zettlemoyer, Luke

PY - 2020/1/1

Y1 - 2020/1/1

N2 - We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation. We also report results for a number of reference models, including a recently state-of-the-art reading comprehension architecture extended to model dialog context. Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data. Dataset, baseline, and leaderboard available at http://quac.ai.

AB - We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation. We also report results for a number of reference models, including a recently state-of-the-art reading comprehension architecture extended to model dialog context. Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data. Dataset, baseline, and leaderboard available at http://quac.ai.

UR - http://www.scopus.com/inward/record.url?scp=85081747763&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85081747763&partnerID=8YFLogxK

M3 - Conference contribution

T3 - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018

SP - 2174

EP - 2184

BT - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018

A2 - Riloff, Ellen

A2 - Chiang, David

A2 - Hockenmaier, Julia

A2 - Tsujii, Jun'ichi

PB - Association for Computational Linguistics

Y2 - 31 October 2018 through 4 November 2018

ER -

QUAC: Question answering in context

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this