TY - GEN
T1 - FedBoost
T2 - 37th International Conference on Machine Learning, ICML 2020
AU - Hamer, Jenny
AU - Mohri, Mehryar
AU - Suresh, Ananda Theertha
N1 - Funding Information:
We warmly thank our colleagues Mingqing Chen, Rajiv Mathews, and Jae Ro for helpful discussions and comments. The work of MM was partly supported by NSF CCF-1535987, NSF IIS-1618662, and a Google Research Award.
Publisher Copyright:
© International Conference on Machine Learning, ICML 2020. All rights reserved.
PY - 2020
Y1 - 2020
N2 - Communication cost is often a bottleneck in federated learning and other client-based distributed learning scenarios. To overcome this, several gradient compression and model compression algorithms have been proposed. In this work, we propose an alternative approach whereby an ensemble of pre-trained base predictors is trained via federated learning. This method allows for training a model which may otherwise surpass the communication bandwidth and storage capacity of the clients to be learned with on-device data through federated learning. Motivated by language modeling, we prove the optimality of ensemble methods for density estimation for standard empirical risk minimization and agnostic risk minimization. We provide communication-efficient ensemble algorithms for federated learning, where per-round communication cost is independent of the size of the ensemble. Furthermore, unlike previous work on gradient compression, our algorithm helps reduce the cost of both server-to-client and client-to-server communication.
AB - Communication cost is often a bottleneck in federated learning and other client-based distributed learning scenarios. To overcome this, several gradient compression and model compression algorithms have been proposed. In this work, we propose an alternative approach whereby an ensemble of pre-trained base predictors is trained via federated learning. This method allows for training a model which may otherwise surpass the communication bandwidth and storage capacity of the clients to be learned with on-device data through federated learning. Motivated by language modeling, we prove the optimality of ensemble methods for density estimation for standard empirical risk minimization and agnostic risk minimization. We provide communication-efficient ensemble algorithms for federated learning, where per-round communication cost is independent of the size of the ensemble. Furthermore, unlike previous work on gradient compression, our algorithm helps reduce the cost of both server-to-client and client-to-server communication.
UR - http://www.scopus.com/inward/record.url?scp=85105271579&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85105271579&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85105271579
T3 - 37th International Conference on Machine Learning, ICML 2020
SP - 3931
EP - 3941
BT - 37th International Conference on Machine Learning, ICML 2020
A2 - Daume, Hal
A2 - Singh, Aarti
PB - International Machine Learning Society (IMLS)
Y2 - 13 July 2020 through 18 July 2020
ER -