NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Sewon Min; Jordan Boyd-Graber; Chris Alberti; Danqi Chen; Eunsol Choi; Michael Collins; Kelvin Guu; Hannaneh Hajishirzi; Kenton Lee; Jennimaria Palomaki; Colin Raffel; Adam Roberts; Tom Kwiatkowski; Patrick Lewis; Yuxiang Wu; Heinrich Küttler; Linqing Liu; Pasquale Minervini; Pontus Stenetorp; Sebastian Riedel; Sohee Yang; Minjoon Seo; Gautier Izacard; Fabio Petroni; Lucas Hosseini; Nicola De Cao; Edouard Grave; Ikuya Yamada; Sonse Shimaoka; Masatoshi Suzuki; Shumpei Miyawaki; Shun Sato; Ryo Takahashi; Jun Suzuki; Martin Fajcik; Martin Docekal; Karel Ondrej; Pavel Smrz; Hao Cheng; Yelong Shen; Xiaodong Liu; Pengcheng He; Weizhu Chen; Jianfeng Gao; Barlas Oguz; Xilun Chen; Vladimir Karpukhin; Stan Peshterliev; Dmytro Okhonko; Michael Schlichtkrull; Sonal Gupta; Yashar Mehdad; Wen Tau Yih

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Küttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian RiedelSohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, Wen Tau Yih

Computer Science

Research output: Contribution to journal › Conference article › peer-review

Abstract

We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.

Original language	English (US)
Pages (from-to)	86-111
Number of pages	26
Journal	Proceedings of Machine Learning Research
Volume	133
State	Published - 2020
Event	34th Demonstration and Competition Track at the 34th Annual Conference on Neural Information Processing Systems, NeurIPS 2020 - Virtual, Online Duration: Dec 6 2020 → Dec 12 2020

Keywords

Knowledge representation
Memory efficiency
Question answering

ASJC Scopus subject areas

Artificial Intelligence
Software
Control and Systems Engineering
Statistics and Probability

Cite this

Min, S., Boyd-Graber, J., Alberti, C., Chen, D., Choi, E., Collins, M., Guu, K., Hajishirzi, H., Lee, K., Palomaki, J., Raffel, C., Roberts, A., Kwiatkowski, T., Lewis, P., Wu, Y., Küttler, H., Liu, L., Minervini, P., Stenetorp, P., ... Yih, W. T. (2020). NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned. Proceedings of Machine Learning Research, 133, 86-111.

Min, S, Boyd-Graber, J, Alberti, C, Chen, D, Choi, E, Collins, M, Guu, K, Hajishirzi, H, Lee, K, Palomaki, J, Raffel, C, Roberts, A, Kwiatkowski, T, Lewis, P, Wu, Y, Küttler, H, Liu, L, Minervini, P, Stenetorp, P, Riedel, S, Yang, S, Seo, M, Izacard, G, Petroni, F, Hosseini, L, De Cao, N, Grave, E, Yamada, I, Shimaoka, S, Suzuki, M, Miyawaki, S, Sato, S, Takahashi, R, Suzuki, J, Fajcik, M, Docekal, M, Ondrej, K, Smrz, P, Cheng, H, Shen, Y, Liu, X, He, P, Chen, W, Gao, J, Oguz, B, Chen, X, Karpukhin, V, Peshterliev, S, Okhonko, D, Schlichtkrull, M, Gupta, S, Mehdad, Y & Yih, WT 2020, 'NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned', Proceedings of Machine Learning Research, vol. 133, pp. 86-111.

@article{09cfe0ff652a444b8fb1e5c66d6b5ac8,

title = "NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned",

abstract = "We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.",

keywords = "Knowledge representation, Memory efficiency, Question answering",

author = "Sewon Min and Jordan Boyd-Graber and Chris Alberti and Danqi Chen and Eunsol Choi and Michael Collins and Kelvin Guu and Hannaneh Hajishirzi and Kenton Lee and Jennimaria Palomaki and Colin Raffel and Adam Roberts and Tom Kwiatkowski and Patrick Lewis and Yuxiang Wu and Heinrich K{\"u}ttler and Linqing Liu and Pasquale Minervini and Pontus Stenetorp and Sebastian Riedel and Sohee Yang and Minjoon Seo and Gautier Izacard and Fabio Petroni and Lucas Hosseini and {De Cao}, Nicola and Edouard Grave and Ikuya Yamada and Sonse Shimaoka and Masatoshi Suzuki and Shumpei Miyawaki and Shun Sato and Ryo Takahashi and Jun Suzuki and Martin Fajcik and Martin Docekal and Karel Ondrej and Pavel Smrz and Hao Cheng and Yelong Shen and Xiaodong Liu and Pengcheng He and Weizhu Chen and Jianfeng Gao and Barlas Oguz and Xilun Chen and Vladimir Karpukhin and Stan Peshterliev and Dmytro Okhonko and Michael Schlichtkrull and Sonal Gupta and Yashar Mehdad and Yih, {Wen Tau}",

note = "Publisher Copyright: {\textcopyright} 2021 S. Min et al.; 34th Demonstration and Competition Track at the 34th Annual Conference on Neural Information Processing Systems, NeurIPS 2020 ; Conference date: 06-12-2020 Through 12-12-2020",

year = "2020",

language = "English (US)",

volume = "133",

pages = "86--111",

journal = "Proceedings of Machine Learning Research",

issn = "2640-3498",

}

TY - JOUR

T1 - NeurIPS 2020 EfficientQA Competition

T2 - 34th Demonstration and Competition Track at the 34th Annual Conference on Neural Information Processing Systems, NeurIPS 2020

AU - Min, Sewon

AU - Boyd-Graber, Jordan

AU - Alberti, Chris

AU - Chen, Danqi

AU - Choi, Eunsol

AU - Collins, Michael

AU - Guu, Kelvin

AU - Hajishirzi, Hannaneh

AU - Lee, Kenton

AU - Palomaki, Jennimaria

AU - Raffel, Colin

AU - Roberts, Adam

AU - Kwiatkowski, Tom

AU - Lewis, Patrick

AU - Wu, Yuxiang

AU - Küttler, Heinrich

AU - Liu, Linqing

AU - Minervini, Pasquale

AU - Stenetorp, Pontus

AU - Riedel, Sebastian

AU - Yang, Sohee

AU - Seo, Minjoon

AU - Izacard, Gautier

AU - Petroni, Fabio

AU - Hosseini, Lucas

AU - De Cao, Nicola

AU - Grave, Edouard

AU - Yamada, Ikuya

AU - Shimaoka, Sonse

AU - Suzuki, Masatoshi

AU - Miyawaki, Shumpei

AU - Sato, Shun

AU - Takahashi, Ryo

AU - Suzuki, Jun

AU - Fajcik, Martin

AU - Docekal, Martin

AU - Ondrej, Karel

AU - Smrz, Pavel

AU - Cheng, Hao

AU - Shen, Yelong

AU - Liu, Xiaodong

AU - He, Pengcheng

AU - Chen, Weizhu

AU - Gao, Jianfeng

AU - Oguz, Barlas

AU - Chen, Xilun

AU - Karpukhin, Vladimir

AU - Peshterliev, Stan

AU - Okhonko, Dmytro

AU - Schlichtkrull, Michael

AU - Gupta, Sonal

AU - Mehdad, Yashar

AU - Yih, Wen Tau

PY - 2020

Y1 - 2020

N2 - We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.

AB - We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.

KW - Knowledge representation

KW - Memory efficiency

KW - Question answering

UR - http://www.scopus.com/inward/record.url?scp=85162627943&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85162627943&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85162627943

SN - 2640-3498

VL - 133

SP - 86

EP - 111

JO - Proceedings of Machine Learning Research

JF - Proceedings of Machine Learning Research

Y2 - 6 December 2020 through 12 December 2020

ER -

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Abstract

Keywords

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this