A fast variational approach for learning Markov random field language models

Yacine Jernite, Alexander M. Rush, David Sontag

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Language modelling is a fundamental building block of natural language processing. However, in practice the size of the vocabulary limits the distributions applicable for this task: specifically, one has to either resort to local optimization methods, such as those used in neural language models, or work with heavily constrained distributions. In this work, we take a step towards overcoming these difficulties. We present a method for global-likelihood optimization of a Markov random field language model exploiting long-range contexts in time independent of the corpus size. We take a variational approach to optimizing the likelihood and exploit underlying symmetries to greatly simplify learning. We demonstrate the efficiency of this method both for language modelling and for part-of-speech tagging.

Original languageEnglish (US)
Title of host publication32nd International Conference on Machine Learning, ICML 2015
EditorsFrancis Bach, David Blei
PublisherInternational Machine Learning Society (IMLS)
Pages2199-2207
Number of pages9
ISBN (Electronic)9781510810587
StatePublished - 2015
Event32nd International Conference on Machine Learning, ICML 2015 - Lile, France
Duration: Jul 6 2015Jul 11 2015

Publication series

Name32nd International Conference on Machine Learning, ICML 2015
Volume3

Other

Other32nd International Conference on Machine Learning, ICML 2015
CountryFrance
CityLile
Period7/6/157/11/15

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Science Applications

Cite this

Jernite, Y., Rush, A. M., & Sontag, D. (2015). A fast variational approach for learning Markov random field language models. In F. Bach, & D. Blei (Eds.), 32nd International Conference on Machine Learning, ICML 2015 (pp. 2199-2207). (32nd International Conference on Machine Learning, ICML 2015; Vol. 3). International Machine Learning Society (IMLS).