Double backpropagation increasing generalization performance

Harris Drucker, Yann Le Cun

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

One test of a new training algorithm is how well the algorithm generalizes from the training data to the test data. It is shown that a new training algorithm termed double backpropagation improves generalization by simultaneously minimizing the normal energy term found in backpropagation and an additional energy term that is related to the sum of the squares of the input derivatives (gradients). In normal backpropagation training, minimizing the energy function tends to push the input gradient to zero. However, this is not always possible. Double backpropagation explicitly pushes the input gradients to zero, making the minimum broader, and increases the generalization on the test data. The authors show the improvement over normal backpropagation on four candidate architectures and a training set of 320 handwritten numbers and a test set of size 180.

Original languageEnglish (US)
Title of host publicationProceedings. IJCNN - International Joint Conference on Neural Networks
Editors Anon
PublisherPubl by IEEE
Pages145-150
Number of pages6
ISBN (Print)0780301641
StatePublished - 1992
EventInternational Joint Conference on Neural Networks - IJCNN-91-Seattle - Seattle, WA, USA
Duration: Jul 8 1991Jul 12 1991

Publication series

NameProceedings. IJCNN - International Joint Conference on Neural Networks

Other

OtherInternational Joint Conference on Neural Networks - IJCNN-91-Seattle
CitySeattle, WA, USA
Period7/8/917/12/91

ASJC Scopus subject areas

  • Engineering(all)

Fingerprint Dive into the research topics of 'Double backpropagation increasing generalization performance'. Together they form a unique fingerprint.

  • Cite this

    Drucker, H., & Le Cun, Y. (1992). Double backpropagation increasing generalization performance. In Anon (Ed.), Proceedings. IJCNN - International Joint Conference on Neural Networks (pp. 145-150). (Proceedings. IJCNN - International Joint Conference on Neural Networks). Publ by IEEE.