TY - GEN
T1 - Two-layer contractive encodings with shortcuts for semi-supervised learning
AU - Schulz, Hannes
AU - Cho, Kyunghyun
AU - Raiko, Tapani
AU - Behnke, Sven
PY - 2013
Y1 - 2013
N2 - Supervised training of multi-layer perceptrons (MLP) with only few labeled examples is prone to overfitting. Pretraining an MLP with unlabeled samples of the input distribution may achieve better generalization. Usually, pretraining is done in a layer-wise, greedy fashion which limits the complexity of the learnable features. To overcome this limitation, two-layer contractive encodings have been proposed recently - which pose a more difficult optimization problem, however. On the other hand, linear transformations of perceptrons have been proposed to make optimization of deep networks easier. In this paper, we propose to combine these two approaches. Experiments on handwritten digit recognition show the benefits of our combined approach to semi-supervised learning.
AB - Supervised training of multi-layer perceptrons (MLP) with only few labeled examples is prone to overfitting. Pretraining an MLP with unlabeled samples of the input distribution may achieve better generalization. Usually, pretraining is done in a layer-wise, greedy fashion which limits the complexity of the learnable features. To overcome this limitation, two-layer contractive encodings have been proposed recently - which pose a more difficult optimization problem, however. On the other hand, linear transformations of perceptrons have been proposed to make optimization of deep networks easier. In this paper, we propose to combine these two approaches. Experiments on handwritten digit recognition show the benefits of our combined approach to semi-supervised learning.
KW - Linear transformation
KW - Multi-layer perceptron
KW - Semi-supervised learning
KW - Two-layer contractive encoding
UR - http://www.scopus.com/inward/record.url?scp=84893365839&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84893365839&partnerID=8YFLogxK
U2 - 10.1007/978-3-642-42054-2_56
DO - 10.1007/978-3-642-42054-2_56
M3 - Conference contribution
AN - SCOPUS:84893365839
SN - 9783642420535
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 450
EP - 457
BT - Neural Information Processing - 20th International Conference, ICONIP 2013, Proceedings
T2 - 20th International Conference on Neural Information Processing, ICONIP 2013
Y2 - 3 November 2013 through 7 November 2013
ER -