The asymptotic spectrum of the Hessian of DNN throughout training

Arthur Jacot; Franck Gabriel; Clément Hongler

The asymptotic spectrum of the Hessian of DNN throughout training

Arthur Jacot, Franck Gabriel, Clément Hongler

Mathematics

Research output: Contribution to conference › Paper › peer-review

Abstract

The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.

Original language	English (US)
State	Published - 2020
Event	8th International Conference on Learning Representations, ICLR 2020 - Addis Ababa, Ethiopia Duration: Apr 30 2020 → …

Conference

Conference	8th International Conference on Learning Representations, ICLR 2020
Country/Territory	Ethiopia
City	Addis Ababa
Period	4/30/20 → …

ASJC Scopus subject areas

Education
Linguistics and Language
Language and Linguistics
Computer Science Applications

Cite this

@conference{bb7fd14360cd48ebaf6c00717e79a5c2,

title = "The asymptotic spectrum of the Hessian of DNN throughout training",

abstract = "The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.",

author = "Arthur Jacot and Franck Gabriel and Cl{\'e}ment Hongler",

note = "Publisher Copyright: {\textcopyright} 2020 8th International Conference on Learning Representations, ICLR 2020. All rights reserved.; 8th International Conference on Learning Representations, ICLR 2020 ; Conference date: 30-04-2020",

year = "2020",

language = "English (US)",

}

TY - CONF

T1 - The asymptotic spectrum of the Hessian of DNN throughout training

AU - Jacot, Arthur

AU - Gabriel, Franck

AU - Hongler, Clément

PY - 2020

Y1 - 2020

N2 - The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.

AB - The dynamics of DNNs during gradient descent is described by the so-called Neural Tangent Kernel (NTK). In this article, we show that the NTK allows one to gain precise insight into the Hessian of the cost of DNNs. When the NTK is fixed during training, we obtain a full characterization of the asymptotics of the spectrum of the Hessian, at initialization and during training. In the so-called mean-field limit, where the NTK is not fixed during training, we describe the first two moments of the Hessian at initialization.

UR - http://www.scopus.com/inward/record.url?scp=85150602165&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85150602165&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85150602165

T2 - 8th International Conference on Learning Representations, ICLR 2020

Y2 - 30 April 2020

ER -

The asymptotic spectrum of the Hessian of DNN throughout training

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this