Collaborative deep learning in fixed topology networks

Zhanhong Jiang, Aditya Balu, Chinmay Hegde, Soumik Sarkar

    Research output: Contribution to journalConference articlepeer-review

    Abstract

    There is significant recent interest to parallelize deep learning algorithms in order to handle the enormous growth in data and model sizes. While most advances focus on model parallelization and engaging multiple computing agents via using a central parameter server, aspect of data parallelization along with decentralized computation has not been explored sufficiently. In this context, this paper presents a new consensus-based distributed SGD (CDSGD) (and its momentum variant, CDMSGD) algorithm for collaborative deep learning over fixed topology networks that enables data parallelization as well as decentralized computation. Such a framework can be extremely useful for learning agents with access to only local/private data in a communication constrained environment. We analyze the convergence properties of the proposed algorithm with strongly convex and nonconvex objective functions with fixed and diminishing step sizes using concepts of Lyapunov function construction. We demonstrate the efficacy of our algorithms in comparison with the baseline centralized SGD and the recently proposed federated averaging algorithm (that also enables data parallelism) based on benchmark datasets such as MNIST, CIFAR-10 and CIFAR-100.

    Original languageEnglish (US)
    Pages (from-to)5905-5915
    Number of pages11
    JournalAdvances in Neural Information Processing Systems
    Volume2017-December
    StatePublished - 2017
    Event31st Annual Conference on Neural Information Processing Systems, NIPS 2017 - Long Beach, United States
    Duration: Dec 4 2017Dec 9 2017

    ASJC Scopus subject areas

    • Computer Networks and Communications
    • Information Systems
    • Signal Processing

    Fingerprint Dive into the research topics of 'Collaborative deep learning in fixed topology networks'. Together they form a unique fingerprint.

    Cite this