Gaussian-Bernoulli deep Boltzmann machine

Kyung Hyun Cho; Tapani Raiko; Alexander Ilin

doi:10.1109/IJCNN.2013.6706831

Gaussian-Bernoulli deep Boltzmann machine

Kyung Hyun Cho, Tapani Raiko, Alexander Ilin

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.

Original language	English (US)
Title of host publication	2013 International Joint Conference on Neural Networks, IJCNN 2013
DOIs	https://doi.org/10.1109/IJCNN.2013.6706831
State	Published - 2013
Event	2013 International Joint Conference on Neural Networks, IJCNN 2013 - Dallas, TX, United States Duration: Aug 4 2013 → Aug 9 2013

Publication series

Name	Proceedings of the International Joint Conference on Neural Networks

Other

Other	2013 International Joint Conference on Neural Networks, IJCNN 2013
Country/Territory	United States
City	Dallas, TX
Period	8/4/13 → 8/9/13

ASJC Scopus subject areas

Software
Artificial Intelligence

Access to Document

10.1109/IJCNN.2013.6706831

Cite this

Cho, KH, Raiko, T & Ilin, A 2013, Gaussian-Bernoulli deep Boltzmann machine. in 2013 International Joint Conference on Neural Networks, IJCNN 2013., 6706831, Proceedings of the International Joint Conference on Neural Networks, 2013 International Joint Conference on Neural Networks, IJCNN 2013, Dallas, TX, United States, 8/4/13. https://doi.org/10.1109/IJCNN.2013.6706831

@inproceedings{af2067d4e6ed4702ae10e02522138cf4,

title = "Gaussian-Bernoulli deep Boltzmann machine",

abstract = "In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.",

author = "Cho, {Kyung Hyun} and Tapani Raiko and Alexander Ilin",

year = "2013",

doi = "10.1109/IJCNN.2013.6706831",

language = "English (US)",

isbn = "9781467361293",

series = "Proceedings of the International Joint Conference on Neural Networks",

booktitle = "2013 International Joint Conference on Neural Networks, IJCNN 2013",

}

TY - GEN

T1 - Gaussian-Bernoulli deep Boltzmann machine

AU - Cho, Kyung Hyun

AU - Raiko, Tapani

AU - Ilin, Alexander

PY - 2013

Y1 - 2013

N2 - In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.

AB - In this paper, we study a model that we call Gaussian-Bernoulli deep Boltzmann machine (GDBM) and discuss potential improvements in training the model. GDBM is designed to be applicable to continuous data and it is constructed from Gaussian-Bernoulli restricted Boltzmann machine (GRBM) by adding multiple layers of binary hidden neurons. The studied improvements of the learning algorithm for GDBM include parallel tempering, enhanced gradient, adaptive learning rate and layer-wise pretraining. We empirically show that they help avoid some of the common difficulties found in training deep Boltzmann machines such as divergence of learning, the difficulty in choosing right learning rate scheduling, and the existence of meaningless higher layers.

UR - http://www.scopus.com/inward/record.url?scp=84893549229&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84893549229&partnerID=8YFLogxK

U2 - 10.1109/IJCNN.2013.6706831

DO - 10.1109/IJCNN.2013.6706831

M3 - Conference contribution

AN - SCOPUS:84893549229

SN - 9781467361293

T3 - Proceedings of the International Joint Conference on Neural Networks

BT - 2013 International Joint Conference on Neural Networks, IJCNN 2013

T2 - 2013 International Joint Conference on Neural Networks, IJCNN 2013

Y2 - 4 August 2013 through 9 August 2013

ER -

Gaussian-Bernoulli deep Boltzmann machine

Abstract

Publication series

Other

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this