Enhanced gradient and adaptive learning rate for training restricted Boltzmann machines

Kyung Hyun Cho, Tapani Raiko, Alexander Ilin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Boltzmann machines are often used as building blocks in greedy learning of deep networks. However, training even a simplified model, known as restricted Boltzmann machine (RBM), can be extremely laborious: Traditional learning algorithms often converge only with the right choice of the learning rate scheduling and the scale of the initial weights. They are also sensitive to specific data representation: An equivalent RBM can be obtained by flipping some bits and changing the weights and biases accordingly, but traditional learning rules are not invariant to such transformations. Without careful tuning of these training settings, traditional algorithms can easily get stuck at plateaus or even diverge. In this work, we present an enhanced gradient which is derived such that it is invariant to bit-flipping transformations. We also propose a way to automatically adjust the learning rate by maximizing a local likelihood estimate. Our experiments confirm that the proposed improvements yield more stable training of RBMs.

Original languageEnglish (US)
Title of host publicationProceedings of the 28th International Conference on Machine Learning, ICML 2011
Pages105-112
Number of pages8
StatePublished - 2011
Event28th International Conference on Machine Learning, ICML 2011 - Bellevue, WA, United States
Duration: Jun 28 2011Jul 2 2011

Publication series

NameProceedings of the 28th International Conference on Machine Learning, ICML 2011

Other

Other28th International Conference on Machine Learning, ICML 2011
CountryUnited States
CityBellevue, WA
Period6/28/117/2/11

ASJC Scopus subject areas

  • Computer Science Applications
  • Human-Computer Interaction
  • Education

Fingerprint Dive into the research topics of 'Enhanced gradient and adaptive learning rate for training restricted Boltzmann machines'. Together they form a unique fingerprint.

Cite this