A unified framework of online learning algorithms for training recurrent neural networks

Owen Marschall, Kyunghyun Cho, Cristina Savin

Research output: Contribution to journalArticlepeer-review

Abstract

We present a framework for compactly summarizing many recent results in efficient and/or biologically plausible online training of recurrent neural networks (RNN). The framework organizes algorithms according to several criteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs. deterministic, and (d) closed form vs. numerical. These axes reveal latent conceptual connections among several recent advances in online learning. Furthermore, we provide novel mathematical intuitions for their degree of success. Testing these algorithms on two parametric task families shows that performances cluster according to our criteria. Although a similar clustering is also observed for pairwise gradient alignment, alignment with exact methods does not explain ultimate performance. This suggests the need for better comparison metrics.

Original languageEnglish (US)
JournalJournal of Machine Learning Research
Volume21
StatePublished - Jun 2020

Keywords

  • Approximation
  • Backpropagation through time
  • Biologically plausible learning
  • Local
  • Online
  • Real-time recurrent learning

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'A unified framework of online learning algorithms for training recurrent neural networks'. Together they form a unique fingerprint.

Cite this