TY - JOUR

T1 - Scaling description of generalization with number of parameters in deep learning

AU - Geiger, Mario

AU - Jacot, Arthur

AU - Spigler, Stefano

AU - Gabriel, Franck

AU - Sagun, Levent

AU - D'Ascoli, Stéphane

AU - Biroli, Giulio

AU - Hongler, Clément

AU - Wyart, Matthieu

N1 - Publisher Copyright:
© 2020 IOP Publishing Ltd and SISSA Medialab srl.

PY - 2020/2

Y1 - 2020/2

N2 - Supervised deep learning involves the training of neural networks with a large number N of parameters. For large enough N, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as N grows past a certain threshold N ∗. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with N. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations fN-(fN) ∼ N-1/4 of the neural net output function fN around its expectation. These affect the generalization error for classification: Under natural assumptions, it decays to a plateau value in a power-law fashion ∼N -1/2. This description breaks down at a so-called jamming transition N = N ∗. At this threshold, we argue that diverges. This result leads to a plausible explanation for the cusp in test error known to occur at N ∗. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond N ∗, and averaging their outputs.

AB - Supervised deep learning involves the training of neural networks with a large number N of parameters. For large enough N, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as N grows past a certain threshold N ∗. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with N. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations fN-(fN) ∼ N-1/4 of the neural net output function fN around its expectation. These affect the generalization error for classification: Under natural assumptions, it decays to a plateau value in a power-law fashion ∼N -1/2. This description breaks down at a so-called jamming transition N = N ∗. At this threshold, we argue that diverges. This result leads to a plausible explanation for the cusp in test error known to occur at N ∗. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond N ∗, and averaging their outputs.

KW - learning theory

KW - machine learning

UR - http://www.scopus.com/inward/record.url?scp=85083155452&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85083155452&partnerID=8YFLogxK

U2 - 10.1088/1742-5468/ab633c

DO - 10.1088/1742-5468/ab633c

M3 - Article

AN - SCOPUS:85083155452

VL - 2020

JO - Journal of Statistical Mechanics: Theory and Experiment

JF - Journal of Statistical Mechanics: Theory and Experiment

SN - 1742-5468

IS - 2

M1 - 023401

ER -