Importance of search and evaluation strategies in neural dialogue modeling

Ilia Kulikov, Alexander H. Miller, Kyunghyun Cho, Jason Weston

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We investigate the impact of search strategies in neural dialogue modeling. We first compare two standard search algorithms, greedy and beam search, as well as our newly proposed iterative beam search which produces a more diverse set of candidate responses. We evaluate these strategies in realistic full conversations with humans and propose a model-based Bayesian calibration to address annotator bias. These conversations are analyzed using two automatic metrics: log-probabilities assigned by the model and utterance diversity. Our experiments reveal that better search algorithms lead to higher rated conversations. However, finding the optimal selection mechanism to choose from a more diverse set of candidates is still an open question.

Original languageEnglish (US)
Title of host publicationINLG 2019 - 12th International Conference on Natural Language Generation, Proceedings of the Conference
PublisherAssociation for Computational Linguistics (ACL)
Pages76-87
Number of pages12
ISBN (Electronic)9781950737949
StatePublished - 2019
Event12th International Conference on Natural Language Generation, INLG 2019 - Tokyo, Japan
Duration: Oct 29 2019Nov 1 2019

Publication series

NameINLG 2019 - 12th International Conference on Natural Language Generation, Proceedings of the Conference

Conference

Conference12th International Conference on Natural Language Generation, INLG 2019
CountryJapan
CityTokyo
Period10/29/1911/1/19

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'Importance of search and evaluation strategies in neural dialogue modeling'. Together they form a unique fingerprint.

Cite this