TY - GEN
T1 - Jump to better conclusions
T2 - 1st Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, co-located with the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
AU - Bastings, Jasmijn
AU - Baroni, Marco
AU - Weston, Jason
AU - Cho, Kyunghyun
AU - Kiela, Douwe
N1 - Funding Information:
We would like to thank Brenden Lake and Marc'Aurelio Ranzato for useful discussions and feedback.
Publisher Copyright:
© 2018 Association for Computational Linguistics
PY - 2018
Y1 - 2018
N2 - Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abilities of recurrent sequence-to-sequence models. Their initial experiments suggested that such models may fail because they lack the ability to extract systematic rules. Here, we take a closer look at SCAN and show that it does not always capture the kind of generalization that it was designed for. To mitigate this we propose a complementary dataset, which requires mapping actions back to the original commands, called NACS. We show that models that do well on SCAN do not necessarily do well on NACS, and that NACS exhibits properties more closely aligned with realistic use-cases for sequence-to-sequence models.
AB - Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abilities of recurrent sequence-to-sequence models. Their initial experiments suggested that such models may fail because they lack the ability to extract systematic rules. Here, we take a closer look at SCAN and show that it does not always capture the kind of generalization that it was designed for. To mitigate this we propose a complementary dataset, which requires mapping actions back to the original commands, called NACS. We show that models that do well on SCAN do not necessarily do well on NACS, and that NACS exhibits properties more closely aligned with realistic use-cases for sequence-to-sequence models.
UR - http://www.scopus.com/inward/record.url?scp=85070349029&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85070349029&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85070349029
T3 - EMNLP 2018 - 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Proceedings of the 1st Workshop
SP - 47
EP - 55
BT - EMNLP 2018 - 2018 EMNLP Workshop BlackboxNLP
PB - Association for Computational Linguistics (ACL)
Y2 - 1 November 2018
ER -