TY - GEN
T1 - Importance of search and evaluation strategies in neural dialogue modeling
AU - Kulikov, Ilia
AU - Miller, Alexander H.
AU - Cho, Kyunghyun
AU - Weston, Jason
N1 - Publisher Copyright:
© 2019 Association for Computational Linguistics
PY - 2019
Y1 - 2019
N2 - We investigate the impact of search strategies in neural dialogue modeling. We first compare two standard search algorithms, greedy and beam search, as well as our newly proposed iterative beam search which produces a more diverse set of candidate responses. We evaluate these strategies in realistic full conversations with humans and propose a model-based Bayesian calibration to address annotator bias. These conversations are analyzed using two automatic metrics: log-probabilities assigned by the model and utterance diversity. Our experiments reveal that better search algorithms lead to higher rated conversations. However, finding the optimal selection mechanism to choose from a more diverse set of candidates is still an open question.
AB - We investigate the impact of search strategies in neural dialogue modeling. We first compare two standard search algorithms, greedy and beam search, as well as our newly proposed iterative beam search which produces a more diverse set of candidate responses. We evaluate these strategies in realistic full conversations with humans and propose a model-based Bayesian calibration to address annotator bias. These conversations are analyzed using two automatic metrics: log-probabilities assigned by the model and utterance diversity. Our experiments reveal that better search algorithms lead to higher rated conversations. However, finding the optimal selection mechanism to choose from a more diverse set of candidates is still an open question.
UR - http://www.scopus.com/inward/record.url?scp=85087163529&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85087163529&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85087163529
T3 - INLG 2019 - 12th International Conference on Natural Language Generation, Proceedings of the Conference
SP - 76
EP - 87
BT - INLG 2019 - 12th International Conference on Natural Language Generation, Proceedings of the Conference
PB - Association for Computational Linguistics (ACL)
T2 - 12th International Conference on Natural Language Generation, INLG 2019
Y2 - 29 October 2019 through 1 November 2019
ER -