Measuring the usefulness of hidden units in Boltzmann machines with mutual information

Mathias Berglund, Tapani Raiko, Kyung Hyun Cho

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results.We confirm experimentally, that the proposed measure is telling how much the performance of the model drops when some of the units of an RBM are pruned away. Our experiments on DBMs highlight differences among different pretraining options.

Original languageEnglish (US)
Title of host publicationNeural Information Processing - 20th International Conference, ICONIP 2013, Proceedings
Pages482-489
Number of pages8
EditionPART 1
DOIs
StatePublished - 2013
Event20th International Conference on Neural Information Processing, ICONIP 2013 - Daegu, Korea, Republic of
Duration: Nov 3 2013Nov 7 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
NumberPART 1
Volume8226 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other20th International Conference on Neural Information Processing, ICONIP 2013
CountryKorea, Republic of
CityDaegu
Period11/3/1311/7/13

Keywords

  • Deep boltzmann machine
  • Deep learning
  • Mutual information
  • Pruning
  • Restricted boltzmann machine
  • Structural learning

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Measuring the usefulness of hidden units in Boltzmann machines with mutual information'. Together they form a unique fingerprint.

  • Cite this

    Berglund, M., Raiko, T., & Cho, K. H. (2013). Measuring the usefulness of hidden units in Boltzmann machines with mutual information. In Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings (PART 1 ed., pp. 482-489). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 8226 LNCS, No. PART 1). https://doi.org/10.1007/978-3-642-42054-2_60