MDPGT: Momentum-Based Decentralized Policy Gradient Tracking

Zhanhong Jiang, Xian Yeow Lee, Sin Yong Tan, Kai Liang Tan, Aditya Balu, Young M. Lee, Chinmay Hegde, Soumik Sarkar

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    We propose a novel policy gradient method for multi-agent reinforcement learning, which leverages two different variancereduction techniques and does not require large batches over iterations. Specifically, we propose a momentum-based decentralized policy gradient tracking (MDPGT) where a new momentum-based variance reduction technique is used to approximate the local policy gradient surrogate with importance sampling, and an intermediate parameter is adopted to track two consecutive policy gradient surrogates. MDPGT provably achieves the best available sample complexity of O(N-1∈-3) for converging to an ∈-stationary point of the global average of N local performance functions (possibly nonconcave). This outperforms the state-of-the-art sample complexity in decentralized model-free reinforcement learning and when initialized with a single trajectory, the sample complexity matches those obtained by the existing decentralized policy gradient methods. We further validate the theoretical claim for the Gaussian policy function. When the required error tolerance ∈ is small enough, MDPGT leads to a linear speed up, which has been previously established in decentralized stochastic optimization, but not for reinforcement learning. Lastly, we provide empirical results on a multi-agent reinforcement learning benchmark environment to support our theoretical findings.

    Original languageEnglish (US)
    Title of host publicationAAAI-22 Technical Tracks 9
    PublisherAssociation for the Advancement of Artificial Intelligence
    Number of pages9
    ISBN (Electronic)1577358767, 9781577358763
    StatePublished - Jun 30 2022
    Event36th AAAI Conference on Artificial Intelligence, AAAI 2022 - Virtual, Online
    Duration: Feb 22 2022Mar 1 2022

    Publication series

    NameProceedings of the 36th AAAI Conference on Artificial Intelligence, AAAI 2022


    Conference36th AAAI Conference on Artificial Intelligence, AAAI 2022
    CityVirtual, Online

    ASJC Scopus subject areas

    • Artificial Intelligence


    Dive into the research topics of 'MDPGT: Momentum-Based Decentralized Policy Gradient Tracking'. Together they form a unique fingerprint.

    Cite this