Swalp: Stochastic weight averaging in low-precision training

Guandao Yang; Tianyi Zhang; Polina Kirichenko; Junwen Bai; Andrew Gordon Wilson; Christopher de Sa

Swalp: Stochastic weight averaging in low-precision training

Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, Christopher de Sa

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.

Original language	English (US)
Title of host publication	36th International Conference on Machine Learning, ICML 2019
Publisher	International Machine Learning Society (IMLS)
Pages	12125-12151
Number of pages	27
ISBN (Electronic)	9781510886988
State	Published - 2019
Event	36th International Conference on Machine Learning, ICML 2019 - Long Beach, United States Duration: Jun 9 2019 → Jun 15 2019

Publication series

Name	36th International Conference on Machine Learning, ICML 2019
Volume	2019-June

Conference

Conference	36th International Conference on Machine Learning, ICML 2019
Country/Territory	United States
City	Long Beach
Period	6/9/19 → 6/15/19

ASJC Scopus subject areas

Education
Computer Science Applications
Human-Computer Interaction

Cite this

Swalp: Stochastic weight averaging in low-precision training. / Yang, Guandao; Zhang, Tianyi; Kirichenko, Polina et al.
36th International Conference on Machine Learning, ICML 2019. International Machine Learning Society (IMLS), 2019. p. 12125-12151 (36th International Conference on Machine Learning, ICML 2019; Vol. 2019-June).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yang, G, Zhang, T, Kirichenko, P, Bai, J, Wilson, AG & de Sa, C 2019, Swalp: Stochastic weight averaging in low-precision training. in 36th International Conference on Machine Learning, ICML 2019. 36th International Conference on Machine Learning, ICML 2019, vol. 2019-June, International Machine Learning Society (IMLS), pp. 12125-12151, 36th International Conference on Machine Learning, ICML 2019, Long Beach, United States, 6/9/19.

@inproceedings{956f5791f54942109d52a49d0fff9990,

title = "Swalp: Stochastic weight averaging in low-precision training",

abstract = "Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.",

author = "Guandao Yang and Tianyi Zhang and Polina Kirichenko and Junwen Bai and Wilson, {Andrew Gordon} and {de Sa}, Christopher",

note = "Funding Information: Polina Kirichenko and Andrew Gordon Wilson were supported by NSF IIS-1563887, an Amazon Research Award, and Facebook Research Award. We thank Google Cloud Platform Research Credits program for providing computational resources. Publisher Copyright: Copyright {\textcopyright} 2019 ASME; 36th International Conference on Machine Learning, ICML 2019 ; Conference date: 09-06-2019 Through 15-06-2019",

year = "2019",

language = "English (US)",

series = "36th International Conference on Machine Learning, ICML 2019",

publisher = "International Machine Learning Society (IMLS)",

pages = "12125--12151",

booktitle = "36th International Conference on Machine Learning, ICML 2019",

}

TY - GEN

T1 - Swalp

T2 - 36th International Conference on Machine Learning, ICML 2019

AU - Yang, Guandao

AU - Zhang, Tianyi

AU - Kirichenko, Polina

AU - Bai, Junwen

AU - Wilson, Andrew Gordon

AU - de Sa, Christopher

N1 - Funding Information: Polina Kirichenko and Andrew Gordon Wilson were supported by NSF IIS-1563887, an Amazon Research Award, and Facebook Research Award. We thank Google Cloud Platform Research Credits program for providing computational resources. Publisher Copyright: Copyright © 2019 ASME

PY - 2019

Y1 - 2019

N2 - Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.

AB - Low precision operations can provide scalability, memory savings, portability, and energy efficiency. This paper proposes SWALP, an approach to low precision training that averages low-precision SGD iterates with a modified learning rate schedule. SWALP is easy to implement and can match the performance of full-precision SGD even with all numbers quantized down to 8 bits, including the gradient accumulators. Additionally, we show that SWALP converges arbitrarily close to the optimal solution for quadratic objectives, and to a noise ball asymptotically smaller than low precision SGD in strongly convex settings.

UR - http://www.scopus.com/inward/record.url?scp=85078283689&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85078283689&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85078283689

T3 - 36th International Conference on Machine Learning, ICML 2019

SP - 12125

EP - 12151

BT - 36th International Conference on Machine Learning, ICML 2019

PB - International Machine Learning Society (IMLS)

Y2 - 9 June 2019 through 15 June 2019

ER -

Swalp: Stochastic weight averaging in low-precision training

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this