TY - GEN
T1 - Implicit regularization of random feature models
AU - Jacot, Arthur
AU - Simsek, Berfin
AU - Spadaro, Francesco
AU - Hongler, Clement
AU - Gabriel, Franck
N1 - Funding Information:
M. Saraiva is financed by Fundo Europeu de Desenvolvimento Regional funds through the COMPETE 2020 Operational Program for Competitiveness and Internationalisation, Portugal 2020, and by Portuguese funds through Fundação para a Ciência e Tecnologia in the framework of the project ‘‘Institute for Research and Innovation in Health Sciences’’ (POCI-01-0145-FEDER-007274), and by Fundação para a Ciência e Tecnologia through Estimulo Individual ao Emprego Científico. P. Vieira is funded by Agence National de la Recherche, through the project MYELOTEN (ANR-13-ISV1-0003-01), and by the Institut Pasteur, France. A. O’Garra is funded by the Francis Crick Institute, which receives its core funding from Cancer Research UK (FC001126), the UK Medical Research Council (FC001126), and the Wellcome Trust (FC001126). The authors declare no competing financial interests.
Funding Information:
M. Saraiva is financed by Fundo Europeu de Desenvolvimento Regional funds through the COMPETE 2020 Operational Program for Competitiveness and Internationalisation, Portugal 2020, and by Portuguese funds through Funda??o para a Ci?ncia e Tecnologia in the framework of the project ??Institute for Research and Innovation in Health Sciences?? (POCI-01-0145-FEDER-007274), and by Funda??o para a Ci?ncia e Tecnologia through Estimulo Individual ao Emprego Cient?fico. P. Vieira is funded by Agence National de la Recherche, through the project MYELOTEN (ANR-13-ISV1-0003-01), and by the Institut Pasteur, France. A. O?Garra is funded by the Francis Crick Institute, which receives its core funding from Cancer Research UK (FC001126), the UK Medical Research Council (FC001126), and the Wellcome Trust (FC001126).
Publisher Copyright:
© International Conference on Machine Learning, ICML 2020. All rights reserved.
PY - 2020
Y1 - 2020
N2 - Random Feature (RF) models are used as efficient parametric approximations of kernel methods. We investigate, by means of random matrix theory, the connection between Gaussian RF models and Kernel Ridge Regression (KRR). For a Gaussian RF model with P features, N data points, and a ridge ?, we show that the average (i.e. expected) RF predictor is close to a KRR predictor with an effective ridge λ. We show that and λ & ? monotonically as P grows, thus revealing the implicit regularization effect of finite RF sampling. We then compare the risk (i.e. test error) of the λ- KRR predictor with the average risk of the ?-RF predictor and obtain a precise and explicit bound on their difference. Finally, we empirically find an extremely good agreement between the test errors of the average ?-RF predictor and λ-KRR predictor.
AB - Random Feature (RF) models are used as efficient parametric approximations of kernel methods. We investigate, by means of random matrix theory, the connection between Gaussian RF models and Kernel Ridge Regression (KRR). For a Gaussian RF model with P features, N data points, and a ridge ?, we show that the average (i.e. expected) RF predictor is close to a KRR predictor with an effective ridge λ. We show that and λ & ? monotonically as P grows, thus revealing the implicit regularization effect of finite RF sampling. We then compare the risk (i.e. test error) of the λ- KRR predictor with the average risk of the ?-RF predictor and obtain a precise and explicit bound on their difference. Finally, we empirically find an extremely good agreement between the test errors of the average ?-RF predictor and λ-KRR predictor.
UR - http://www.scopus.com/inward/record.url?scp=85105261763&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85105261763&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85105261763
T3 - 37th International Conference on Machine Learning, ICML 2020
SP - 4581
EP - 4590
BT - 37th International Conference on Machine Learning, ICML 2020
A2 - Daume, Hal
A2 - Singh, Aarti
PB - International Machine Learning Society (IMLS)
T2 - 37th International Conference on Machine Learning, ICML 2020
Y2 - 13 July 2020 through 18 July 2020
ER -