Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks

João Loula, Marco Baroni, Brenden M. Lake

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Systematic compositionality is the ability to recombine meaningful units with regular and predictable outcomes, and it's seen as key to the human capacity for generalization in language. Recent work (Lake and Baroni, 2018) has studied systematic compositionality in modern seq2seq models using generalization to novel navigation instructions in a grounded environment as a probing tool. Lake and Baroni's main experiment required the models to quickly bootstrap the meaning of new words. We extend this framework here to settings where the model needs only to recombine well-trained functional words (such as “around” and “right”) in novel contexts. Our findings confirm and strengthen the earlier ones: seq2seq models can be impressively good at generalizing to novel combinations of previously-seen input, but only when they receive extensive training on the specific pattern to be generalized (e.g., generalizing from many examples of “X around right” to “jump around right”), while failing when generalization requires novel application of compositional rules (e.g., inferring the meaning of “around right” from those of “right” and “around”).

Original languageEnglish (US)
Title of host publicationEMNLP 2018 - 2018 EMNLP Workshop BlackboxNLP
Subtitle of host publicationAnalyzing and Interpreting Neural Networks for NLP, Proceedings of the 1st Workshop
PublisherAssociation for Computational Linguistics (ACL)
Pages108-114
Number of pages7
ISBN (Electronic)9781948087711
StatePublished - 2018
Event1st Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, co-located with the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018 - Brussels, Belgium
Duration: Nov 1 2018 → …

Publication series

NameEMNLP 2018 - 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Proceedings of the 1st Workshop

Conference

Conference1st Workshop on BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, co-located with the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018
Country/TerritoryBelgium
CityBrussels
Period11/1/18 → …

ASJC Scopus subject areas

  • Computer Science Applications
  • Computational Theory and Mathematics
  • Information Systems

Fingerprint

Dive into the research topics of 'Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks'. Together they form a unique fingerprint.

Cite this