TY - GEN
T1 - Measuring the usefulness of hidden units in Boltzmann machines with mutual information
AU - Berglund, Mathias
AU - Raiko, Tapani
AU - Cho, Kyung Hyun
N1 - Funding Information:
This work was supported by the Academy of Finland (Finnish Centre of Excellence in Computational Inference Research COIN, 251170 ).
PY - 2013
Y1 - 2013
N2 - Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results.We confirm experimentally, that the proposed measure is telling how much the performance of the model drops when some of the units of an RBM are pruned away. Our experiments on DBMs highlight differences among different pretraining options.
AB - Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in deep learning, but it is often difficult to measure their performance in general, or measure the importance of individual hidden units in specific. We propose to use mutual information to measure the usefulness of individual hidden units in Boltzmann machines. The measure serves as an upper bound for the information the neuron can pass on, enabling detection of a particular kind of poor training results.We confirm experimentally, that the proposed measure is telling how much the performance of the model drops when some of the units of an RBM are pruned away. Our experiments on DBMs highlight differences among different pretraining options.
KW - Deep boltzmann machine
KW - Deep learning
KW - Mutual information
KW - Pruning
KW - Restricted boltzmann machine
KW - Structural learning
UR - http://www.scopus.com/inward/record.url?scp=84893348709&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84893348709&partnerID=8YFLogxK
U2 - 10.1007/978-3-642-42054-2_60
DO - 10.1007/978-3-642-42054-2_60
M3 - Conference contribution
AN - SCOPUS:84893348709
SN - 9783642420535
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 482
EP - 489
BT - Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings
T2 - 20th International Conference on Neural Information Processing, ICONIP 2013
Y2 - 3 November 2013 through 7 November 2013
ER -