Tikhonov-type regularization for restricted Boltzmann machines

Kyung Hyun Cho; Alexander Ilin; Tapani Raiko

doi:10.1007/978-3-642-33269-2_11

Tikhonov-type regularization for restricted Boltzmann machines

Kyung Hyun Cho, Alexander Ilin, Tapani Raiko

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.

Original language	English (US)
Title of host publication	Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings
Pages	81-88
Number of pages	8
Edition	PART 1
DOIs	https://doi.org/10.1007/978-3-642-33269-2_11
State	Published - 2012
Event	22nd International Conference on Artificial Neural Networks, ICANN 2012 - Lausanne, Switzerland Duration: Sep 11 2012 → Sep 14 2012

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Number	PART 1
Volume	7552 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Other

Other	22nd International Conference on Artificial Neural Networks, ICANN 2012
Country/Territory	Switzerland
City	Lausanne
Period	9/11/12 → 9/14/12

Keywords

Restricted Boltzmann Machine
Tikhonov Regularization

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-642-33269-2_11

Cite this

Cho, K. H., Ilin, A., & Raiko, T. (2012). Tikhonov-type regularization for restricted Boltzmann machines. In Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings (PART 1 ed., pp. 81-88). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7552 LNCS, No. PART 1). https://doi.org/10.1007/978-3-642-33269-2_11

Tikhonov-type regularization for restricted Boltzmann machines. / Cho, Kyung Hyun; Ilin, Alexander; Raiko, Tapani.
Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings. PART 1. ed. 2012. p. 81-88 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7552 LNCS, No. PART 1).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Cho, KH, Ilin, A & Raiko, T 2012, Tikhonov-type regularization for restricted Boltzmann machines. in Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings. PART 1 edn, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), no. PART 1, vol. 7552 LNCS, pp. 81-88, 22nd International Conference on Artificial Neural Networks, ICANN 2012, Lausanne, Switzerland, 9/11/12. https://doi.org/10.1007/978-3-642-33269-2_11

Cho KH, Ilin A, Raiko T. Tikhonov-type regularization for restricted Boltzmann machines. In Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings. PART 1 ed. 2012. p. 81-88. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1). doi: 10.1007/978-3-642-33269-2_11

Cho, Kyung Hyun ; Ilin, Alexander ; Raiko, Tapani. / Tikhonov-type regularization for restricted Boltzmann machines. Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings. PART 1. ed. 2012. pp. 81-88 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); PART 1).

@inproceedings{55b6ba6a6ce34b76b83afefec51024a9,

title = "Tikhonov-type regularization for restricted Boltzmann machines",

abstract = "In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.",

keywords = "Restricted Boltzmann Machine, Tikhonov Regularization",

author = "Cho, {Kyung Hyun} and Alexander Ilin and Tapani Raiko",

year = "2012",

doi = "10.1007/978-3-642-33269-2_11",

language = "English (US)",

isbn = "9783642332685",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

number = "PART 1",

pages = "81--88",

booktitle = "Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings",

edition = "PART 1",

note = "22nd International Conference on Artificial Neural Networks, ICANN 2012 ; Conference date: 11-09-2012 Through 14-09-2012",

}

TY - GEN

T1 - Tikhonov-type regularization for restricted Boltzmann machines

AU - Cho, Kyung Hyun

AU - Ilin, Alexander

AU - Raiko, Tapani

PY - 2012

Y1 - 2012

N2 - In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.

AB - In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both formulations turn out to be combinations of the widely used weight-decay and sparsity regularization. We empirically evaluate the effect of the proposed regularization schemes and show that the use of them could help extracting better discriminative features with sparser hidden activation probabilities.

KW - Restricted Boltzmann Machine

KW - Tikhonov Regularization

UR - http://www.scopus.com/inward/record.url?scp=84867671137&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84867671137&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-33269-2_11

DO - 10.1007/978-3-642-33269-2_11

M3 - Conference contribution

AN - SCOPUS:84867671137

SN - 9783642332685

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 81

EP - 88

BT - Artificial Neural Networks and Machine Learning, ICANN 2012 - 22nd International Conference on Artificial Neural Networks, Proceedings

T2 - 22nd International Conference on Artificial Neural Networks, ICANN 2012

Y2 - 11 September 2012 through 14 September 2012

ER -

Tikhonov-type regularization for restricted Boltzmann machines

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this