The role of over-parametrization in generalization of neural networks

Behnam Neyshabur; Zhiyuan Li; Srinadh Bhojanapalli; Yann LeCun; Nathan Srebro

The role of over-parametrization in generalization of neural networks

Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, Nathan Srebro

Computer Science

Research output: Contribution to conference › Paper › peer-review

Abstract

Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes (within the range reported in the experiments), and could partly explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

Original language	English (US)
State	Published - Jan 1 2019
Event	7th International Conference on Learning Representations, ICLR 2019 - New Orleans, United States Duration: May 6 2019 → May 9 2019

Conference

Conference	7th International Conference on Learning Representations, ICLR 2019
Country/Territory	United States
City	New Orleans
Period	5/6/19 → 5/9/19

ASJC Scopus subject areas

Education
Computer Science Applications
Linguistics and Language
Language and Linguistics

Cite this

@conference{65e684e59417479dbe3fd97c22618b2c,

title = "The role of over-parametrization in generalization of neural networks",

abstract = "Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes (within the range reported in the experiments), and could partly explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.",

author = "Behnam Neyshabur and Zhiyuan Li and Srinadh Bhojanapalli and Yann LeCun and Nathan Srebro",

note = "Funding Information: The authors thank Sanjeev Arora for many fruitful discussions on generalization of neural networks and David McAllester for discussion on the distance to random initialization. The authors also thank Wei Zhan for discussion on the lower bound. This research was supported in part by NSF IIS-RI award 1302662 and Schmidt Foundation. Publisher Copyright: {\textcopyright} 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved.; 7th International Conference on Learning Representations, ICLR 2019 ; Conference date: 06-05-2019 Through 09-05-2019",

year = "2019",

month = jan,

day = "1",

language = "English (US)",

}

TY - CONF

T1 - The role of over-parametrization in generalization of neural networks

AU - Neyshabur, Behnam

AU - Li, Zhiyuan

AU - Bhojanapalli, Srinadh

AU - LeCun, Yann

AU - Srebro, Nathan

N1 - Funding Information: The authors thank Sanjeev Arora for many fruitful discussions on generalization of neural networks and David McAllester for discussion on the distance to random initialization. The authors also thank Wei Zhan for discussion on the lower bound. This research was supported in part by NSF IIS-RI award 1302662 and Schmidt Foundation. Publisher Copyright: © 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved.

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes (within the range reported in the experiments), and could partly explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

AB - Despite existing work on ensuring generalization of neural networks in terms of scale sensitive complexity measures, such as norms, margin and sharpness, these complexity measures do not offer an explanation of why neural networks generalize better with over-parametrization. In this work we suggest a novel complexity measure based on unit-wise capacities resulting in a tighter generalization bound for two layer ReLU networks. Our capacity bound correlates with the behavior of test error with increasing network sizes (within the range reported in the experiments), and could partly explain the improvement in generalization with over-parametrization. We further present a matching lower bound for the Rademacher complexity that improves over previous capacity lower bounds for neural networks.

UR - http://www.scopus.com/inward/record.url?scp=85071150465&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071150465&partnerID=8YFLogxK

M3 - Paper

T2 - 7th International Conference on Learning Representations, ICLR 2019

Y2 - 6 May 2019 through 9 May 2019

ER -

The role of over-parametrization in generalization of neural networks

Abstract

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this