TY - JOUR

T1 - Neural tangent kernel

T2 - 32nd Conference on Neural Information Processing Systems, NeurIPS 2018

AU - Jacot, Arthur

AU - Gabriel, Franck

AU - Hongler, Clément

N1 - Funding Information:
The authors thank K. Kytölä for many interesting discussions. The second author was supported by the ERC CG CRITICAL. The last author acknowledges support from the ERC SG Constamis, the NCCR SwissMAP, the Blavatnik Family Foundation and the Latsis Foundation.
Publisher Copyright:
© 2018 Curran Associates Inc.All rights reserved.

PY - 2018

Y1 - 2018

N2 - At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit (12; 9), thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function fθ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function fθ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

AB - At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit (12; 9), thus connecting them to kernel methods. We prove that the evolution of an ANN during training can also be described by a kernel: during gradient descent on the parameters of an ANN, the network function fθ (which maps input vectors to output vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK). This kernel is central to describe the generalization features of ANNs. While the NTK is random at initialization and varies during training, in the infinite-width limit it converges to an explicit limiting kernel and it stays constant during training. This makes it possible to study the training of ANNs in function space instead of parameter space. Convergence of the training can then be related to the positive-definiteness of the limiting NTK. We then focus on the setting of least-squares regression and show that in the infinite-width limit, the network function fθ follows a linear differential equation during training. The convergence is fastest along the largest kernel principal components of the input data with respect to the NTK, hence suggesting a theoretical motivation for early stopping. Finally we study the NTK numerically, observe its behavior for wide networks, and compare it to the infinite-width limit.

UR - http://www.scopus.com/inward/record.url?scp=85064845355&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064845355&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85064845355

SN - 1049-5258

VL - 2018-December

SP - 8571

EP - 8580

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

Y2 - 2 December 2018 through 8 December 2018

ER -