Efficient backprop

Yann A. LeCun, Léon Bottou, Genevieve B. Orr, Klaus Robert Müller

Research output: Chapter in Book/Report/Conference proceedingChapter

Abstract

The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most "classical" second-order methods are impractical for large neural networks. A few methods are proposed that do not have these limitations.

Original languageEnglish (US)
Title of host publicationNeural Networks
Subtitle of host publicationTricks of the Trade
EditorsGregoire Montavon, Klaus-Robert Muller, Genevieve B. Orr, Klaus-Robert Muller
Pages9-48
Number of pages40
DOIs
StatePublished - 2012

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume7700 LECTURE NO
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Efficient backprop'. Together they form a unique fingerprint.

Cite this