Generating sentences from a continuous space

Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, Samy Bengio

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    The standard recurrent neural network language model (rnnlm) generates sentences one word at a time and does not work from an explicit global sentence representation. In this work, we introduce and study an rnn-based variational autoencoder generative model that incorporates distributed latent representations of entire sentences. This factorization allows it to explicitly model holistic properties of sentences such as style, topic, and high-level syntactic features. Samples from the prior over these sentence representations remarkably produce diverse and well-formed sentences through simple deterministic decoding. By examining paths through this latent space, we are able to generate coherent novel sentences that interpolate between known sentences. We present techniques for solving the difficult learning problem presented by this model, demonstrate its effectiveness in imputing missing words, explore many interesting properties of the model’s latent sentence space, and present negative results on the use of the model in language modeling.

    Original languageEnglish (US)
    Title of host publicationCoNLL 2016 - 20th SIGNLL Conference on Computational Natural Language Learning, Proceedings
    PublisherAssociation for Computational Linguistics (ACL)
    Pages10-21
    Number of pages12
    ISBN (Electronic)9781945626197
    StatePublished - Jan 1 2016
    Event20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016 - Berlin, Germany
    Duration: Aug 11 2016Aug 12 2016

    Publication series

    NameCoNLL 2016 - 20th SIGNLL Conference on Computational Natural Language Learning, Proceedings

    Conference

    Conference20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016
    CountryGermany
    CityBerlin
    Period8/11/168/12/16

    ASJC Scopus subject areas

    • Artificial Intelligence
    • Human-Computer Interaction
    • Linguistics and Language

    Fingerprint Dive into the research topics of 'Generating sentences from a continuous space'. Together they form a unique fingerprint.

  • Cite this

    Bowman, S. R., Vilnis, L., Vinyals, O., Dai, A. M., Jozefowicz, R., & Bengio, S. (2016). Generating sentences from a continuous space. In CoNLL 2016 - 20th SIGNLL Conference on Computational Natural Language Learning, Proceedings (pp. 10-21). (CoNLL 2016 - 20th SIGNLL Conference on Computational Natural Language Learning, Proceedings). Association for Computational Linguistics (ACL).