Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

Yangkun Chen; Chenghui Yu; Hengman Zhu; Shuai Liu; Yibing Zhang; Julian Togelius; Xiu Li; Joseph Suarez; Bo Wu; Rui Du; Weijun Hong; Liang Zhao; Sharada Mohanty; Xiaolong Zhu; Junjie Zhang; Hanmo Chen; Shanliang Qian; Jinke He; Clare Zhu; Jiaxin Chen; Phillip Isola

Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

Yangkun Chen, Chenghui Yu, Hengman Zhu, Shuai Liu, Yibing Zhang, Julian Togelius, Xiu Li, Joseph Suarez, Bo Wu, Rui Du, Weijun Hong, Liang Zhao, Sharada Mohanty, Xiaolong Zhu, Junjie Zhang, Hanmo Chen, Shanliang Qian, Jinke He, Clare Zhu, Jiaxin ChenPhillip Isola

Research output: Contribution to journal › Conference article › peer-review

Abstract

We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

Original language	English (US)
Pages (from-to)	2490-2492
Number of pages	3
Journal	Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS
Volume	2023-May
State	Published - 2023
Event	22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023 - London, United Kingdom Duration: May 29 2023 → Jun 2 2023

Keywords

Benchmark
Competition
Multi-agent Reinforcement Learning

ASJC Scopus subject areas

Artificial Intelligence
Software
Control and Systems Engineering

Cite this

Chen, Y., Yu, C., Zhu, H., Liu, S., Zhang, Y., Togelius, J., Li, X., Suarez, J., Wu, B., Du, R., Hong, W., Zhao, L., Mohanty, S., Zhu, X., Zhang, J., Chen, H., Qian, S., He, J., Zhu, C., ... Isola, P. (2023). Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO. Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2023-May, 2490-2492.

Chen, Y, Yu, C, Zhu, H, Liu, S, Zhang, Y, Togelius, J, Li, X, Suarez, J, Wu, B, Du, R, Hong, W, Zhao, L, Mohanty, S, Zhu, X, Zhang, J, Chen, H, Qian, S, He, J, Zhu, C, Chen, J & Isola, P 2023, 'Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO', Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, vol. 2023-May, pp. 2490-2492.

@article{228ba2fadc6a4665b2ab1535250cd27c,

title = "Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO",

abstract = "We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.",

keywords = "Benchmark, Competition, Multi-agent Reinforcement Learning",

author = "Yangkun Chen and Chenghui Yu and Hengman Zhu and Shuai Liu and Yibing Zhang and Julian Togelius and Xiu Li and Joseph Suarez and Bo Wu and Rui Du and Weijun Hong and Liang Zhao and Sharada Mohanty and Xiaolong Zhu and Junjie Zhang and Hanmo Chen and Shanliang Qian and Jinke He and Clare Zhu and Jiaxin Chen and Phillip Isola",

note = "Publisher Copyright: {\textcopyright} 2023 International Foundation for Autonomous Agents and Multiagent Systems (www.ifaamas.org). All rights reserved.; 22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023 ; Conference date: 29-05-2023 Through 02-06-2023",

year = "2023",

language = "English (US)",

volume = "2023-May",

pages = "2490--2492",

journal = "Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS",

issn = "1548-8403",

publisher = "International Foundation for Autonomous Agents and Multiagent Systems (IFAAMAS)",

}

TY - JOUR

T1 - Benchmarking Robustness and Generalization in Multi-Agent Systems

T2 - 22nd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023

AU - Chen, Yangkun

AU - Yu, Chenghui

AU - Zhu, Hengman

AU - Liu, Shuai

AU - Zhang, Yibing

AU - Togelius, Julian

AU - Li, Xiu

AU - Suarez, Joseph

AU - Wu, Bo

AU - Du, Rui

AU - Hong, Weijun

AU - Zhao, Liang

AU - Mohanty, Sharada

AU - Zhu, Xiaolong

AU - Zhang, Junjie

AU - Chen, Hanmo

AU - Qian, Shanliang

AU - He, Jinke

AU - Zhu, Clare

AU - Chen, Jiaxin

AU - Isola, Phillip

PY - 2023

Y1 - 2023

N2 - We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

AB - We present the results of the second Neural MMO challenge, hosted at IJCAI 2022, which received 1600+ submissions. This competition targets robustness and generalization in multi-agent systems: participants train teams of agents to complete a multi-task objective against opponents not seen during training. We summarize the competition design and results and suggest that, considering our work as a case study, competitions are an effective approach to solving hard problems and establishing a solid benchmark for algorithms. We will open-source our benchmark including the environment wrapper, baselines, a visualization tool, and selected policies for further research.

KW - Benchmark

KW - Competition

KW - Multi-agent Reinforcement Learning

UR - http://www.scopus.com/inward/record.url?scp=85171258676&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85171258676&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85171258676

SN - 1548-8403

VL - 2023-May

SP - 2490

EP - 2492

JO - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS

JF - Proceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS

Y2 - 29 May 2023 through 2 June 2023

ER -

Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMO

Abstract

Keywords

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this