Fast convolutional nets with fbfft: A GPU performance evaluation

Nicolas Vasilache; Jeff Johnson; Michael Mathieu; Soumith Chintala; Serkan Piantino; Yann LeCun

Fast convolutional nets with fbfft: A GPU performance evaluation

Nicolas Vasilache, Jeff Johnson, Michael Mathieu, Soumith Chintala, Serkan Piantino, Yann LeCun

Computer Science

Research output: Contribution to conference › Paper › peer-review

Abstract

We examine the performance profile of Convolutional Neural Network (CNN) training on the current generation of NVIDIA Graphics Processing Units (GPUs). We introduce two new Fast Fourier Transform convolution implementations: one based on NVIDIA’s cuFFT library, and another based on a Facebook authored FFT implementation, fbfft, that provides significant speedups over cuFFT (over 1.5 x) for whole CNNs. Both of these convolution implementations are available in open source, and are faster than NVIDIA’s cuDNN implementation for many common convolutional layers (up to 23.5 x for a synthetic kernel configuration). We discuss different performance regimes of convolutions, comparing areas where straightforward time domain convolutions outperform Fourier frequency domain convolutions. Details on algorithmic applications of NVIDIA GPU hardware specifics in the implementation of fbfft are also provided.

Original language	English (US)
State	Published - Jan 1 2015
Event	3rd International Conference on Learning Representations, ICLR 2015 - San Diego, United States Duration: May 7 2015 → May 9 2015

Conference

Conference	3rd International Conference on Learning Representations, ICLR 2015
Country/Territory	United States
City	San Diego
Period	5/7/15 → 5/9/15

ASJC Scopus subject areas

Education
Computer Science Applications
Linguistics and Language
Language and Linguistics

Cite this

@conference{c20bbfea5e7f4e7e9467e576138ebbc4,

title = "Fast convolutional nets with fbfft: A GPU performance evaluation",

abstract = "We examine the performance profile of Convolutional Neural Network (CNN) training on the current generation of NVIDIA Graphics Processing Units (GPUs). We introduce two new Fast Fourier Transform convolution implementations: one based on NVIDIA{\textquoteright}s cuFFT library, and another based on a Facebook authored FFT implementation, fbfft, that provides significant speedups over cuFFT (over 1.5 x) for whole CNNs. Both of these convolution implementations are available in open source, and are faster than NVIDIA{\textquoteright}s cuDNN implementation for many common convolutional layers (up to 23.5 x for a synthetic kernel configuration). We discuss different performance regimes of convolutions, comparing areas where straightforward time domain convolutions outperform Fourier frequency domain convolutions. Details on algorithmic applications of NVIDIA GPU hardware specifics in the implementation of fbfft are also provided.",

author = "Nicolas Vasilache and Jeff Johnson and Michael Mathieu and Soumith Chintala and Serkan Piantino and Yann LeCun",

year = "2015",

month = jan,

day = "1",

language = "English (US)",

note = "3rd International Conference on Learning Representations, ICLR 2015 ; Conference date: 07-05-2015 Through 09-05-2015",

}

TY - CONF

T1 - Fast convolutional nets with fbfft

T2 - 3rd International Conference on Learning Representations, ICLR 2015

AU - Vasilache, Nicolas

AU - Johnson, Jeff

AU - Mathieu, Michael

AU - Chintala, Soumith

AU - Piantino, Serkan

AU - LeCun, Yann

PY - 2015/1/1

Y1 - 2015/1/1

N2 - We examine the performance profile of Convolutional Neural Network (CNN) training on the current generation of NVIDIA Graphics Processing Units (GPUs). We introduce two new Fast Fourier Transform convolution implementations: one based on NVIDIA’s cuFFT library, and another based on a Facebook authored FFT implementation, fbfft, that provides significant speedups over cuFFT (over 1.5 x) for whole CNNs. Both of these convolution implementations are available in open source, and are faster than NVIDIA’s cuDNN implementation for many common convolutional layers (up to 23.5 x for a synthetic kernel configuration). We discuss different performance regimes of convolutions, comparing areas where straightforward time domain convolutions outperform Fourier frequency domain convolutions. Details on algorithmic applications of NVIDIA GPU hardware specifics in the implementation of fbfft are also provided.

AB - We examine the performance profile of Convolutional Neural Network (CNN) training on the current generation of NVIDIA Graphics Processing Units (GPUs). We introduce two new Fast Fourier Transform convolution implementations: one based on NVIDIA’s cuFFT library, and another based on a Facebook authored FFT implementation, fbfft, that provides significant speedups over cuFFT (over 1.5 x) for whole CNNs. Both of these convolution implementations are available in open source, and are faster than NVIDIA’s cuDNN implementation for many common convolutional layers (up to 23.5 x for a synthetic kernel configuration). We discuss different performance regimes of convolutions, comparing areas where straightforward time domain convolutions outperform Fourier frequency domain convolutions. Details on algorithmic applications of NVIDIA GPU hardware specifics in the implementation of fbfft are also provided.

UR - http://www.scopus.com/inward/record.url?scp=85070891854&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85070891854&partnerID=8YFLogxK

M3 - Paper

Y2 - 7 May 2015 through 9 May 2015

ER -

Fast convolutional nets with fbfft: A GPU performance evaluation

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this