Leveling the playing field: Fairness in AI versus human game benchmarks

Rodrigo Canaan, Christoph Salge, Julian Togelius, Andy Nealen

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    From the beginning of the history of AI, there has been interest in games as a platform of research. As the field developed, human-level competence in complex games became a target researchers worked to reach. Only relatively recently has this target been finally met for traditional tabletop games such as Backgammon, Chess and Go. This prompted a shift in research focus towards electronic games, which provide unique new challenges. As is often the case with AI research, these results are liable to be exaggerated or mis-represented by either authors or third parties. The extent to which these game benchmarks constitute "fair" competition between human and AI is also a matter of debate. In this paper, we review statements made by reseachers and third parties in the general media and academic publications about these game benchmark results. We analyze what a fair competition would look like and suggest a taxonomy of dimensions to frame the debate of fairness in game contests between humans and machines. Eventually, we argue that there is no completely fair way to compare human and AI performance on a game.

    Original languageEnglish (US)
    Title of host publicationProceedings of the 14th International Conference on the Foundations of Digital Games, FDG 2019
    EditorsFoaad Khosmood, Johanna Pirker, Thomas Apperley, Sebastian Deterding
    PublisherAssociation for Computing Machinery
    ISBN (Electronic)9781450372176
    DOIs
    StatePublished - Aug 26 2019
    Event14th International Conference on the Foundations of Digital Games, FDG 2019 - San Luis Obispo, United States
    Duration: Aug 26 2019Aug 30 2019

    Publication series

    NameACM International Conference Proceeding Series

    Conference

    Conference14th International Conference on the Foundations of Digital Games, FDG 2019
    CountryUnited States
    CitySan Luis Obispo
    Period8/26/198/30/19

    Keywords

    • AI benchmarks
    • Fairness
    • Game AI
    • games

    ASJC Scopus subject areas

    • Software
    • Human-Computer Interaction
    • Computer Vision and Pattern Recognition
    • Computer Networks and Communications

    Fingerprint Dive into the research topics of 'Leveling the playing field: Fairness in AI versus human game benchmarks'. Together they form a unique fingerprint.

  • Cite this

    Canaan, R., Salge, C., Togelius, J., & Nealen, A. (2019). Leveling the playing field: Fairness in AI versus human game benchmarks. In F. Khosmood, J. Pirker, T. Apperley, & S. Deterding (Eds.), Proceedings of the 14th International Conference on the Foundations of Digital Games, FDG 2019 [37] (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3337722.3337750