Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

Yangkun Chen, Chenghui Yu, Hengman Zhu, Shuai Liu, Yibing Zhang, Julian Togelius, Xiu Li, Joseph Suarez, Bo Wu, Rui Du, Weijun Hong, Liang Zhao, Sharada Mohanty, Xiaolong Zhu, Junjie Zhang, Hanmo Chen, Shanliang Qian, Jinke He, Clare Zhu, Jiaxin ChenPhillip Isola

    Research output: Contribution to journalConference articlepeer-review

    Abstract

    We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

    Original languageEnglish (US)
    Pages (from-to)2490-2492
    Number of pages3
    JournalProceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
    Volume2023-May
    StatePublished - 2023
    Event22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023 - London, United Kingdom
    Duration: May 29 2023Jun 2 2023

    Keywords

    • Benchmark
    • Competition
    • Multi-agent Reinforcement Learning

    ASJC Scopus subject areas

    • Artificial Intelligence
    • Software
    • Control and Systems Engineering

    Fingerprint

    Dive into the research topics of 'Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO'. Together they form a unique fingerprint.

    Cite this