Pushing stochastic gradient towards second-order methods – backpropagation learning with transformations in nonlinearities

Tommi Vatanen, Tapani Raiko, Harri Valpola, Yann LeCun

Research output: Contribution to conferencePaper

Abstract

Recently, we proposed to transform the outputs of each hidden neuron in a multilayer perceptron network to have zero output and zero slope on average, and use separate shortcut connections to model the linear dependencies instead. We continue the work by firstly introducing a third transformation to normalize the scale of the outputs of each hidden neuron, and secondly by analyzing the connections to second order optimization methods. We show that the transformations make a simple stochastic gradient behave closer to second-order optimization methods and thus speed up learning. This is shown both in theory and with experiments. The experiments on the third transformation show that while it further increases the speed of learning, it can also hurt performance by converging to a worse local optimum, where both the inputs and outputs of many hidden neurons are close to zero.

Original languageEnglish (US)
StatePublished - Jan 1 2013
Event1st International Conference on Learning Representations, ICLR 2013 - Scottsdale, United States
Duration: May 2 2013May 4 2013

Conference

Conference1st International Conference on Learning Representations, ICLR 2013
CountryUnited States
CityScottsdale
Period5/2/135/4/13

ASJC Scopus subject areas

  • Education
  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint Dive into the research topics of 'Pushing stochastic gradient towards second-order methods – backpropagation learning with transformations in nonlinearities'. Together they form a unique fingerprint.

  • Cite this

    Vatanen, T., Raiko, T., Valpola, H., & LeCun, Y. (2013). Pushing stochastic gradient towards second-order methods – backpropagation learning with transformations in nonlinearities. Paper presented at 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, United States.