Scaling, Control and Generalization in Reinforcement Learning Level Generators

Sam Earle, Zehua Jiang, Julian Togelius

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Procedural Content Generation via Reinforcement Learning (PCGRL) has been introduced as a means by which controllable designer agents can be trained based only on a set of computable metrics acting as a proxy for the level's quality and key characteristics. While PCGRL offers a unique set of affordances for game designers, it is constrained by the compute-intensive process of training RL agents, and has so far been limited to generating relatively small levels. To address this issue of scale, we implement several PCGRL environments in Jax so that all aspects of learning and simulation happen in parallel on the GPU, resulting in faster environment simulation; removing the CPU-GPU transfer of information bottleneck during RL training; and ultimately resulting in significantly improved training speed. We replicate several key results from prior works in this new framework, letting models train for much longer than previously studied, and evaluating their behavior after 1 billion timesteps. Aiming for greater control for human designers, we introduce randomized level sizes and frozen 'pinpoints' of pivotal game tiles as further ways of countering overfitting. To test the generalization ability of learned generators, we evaluate models on large, out-of-distribution map sizes, and find that partial observation sizes learn more robust design strategies.

    Original languageEnglish (US)
    Title of host publicationProceedings of the 2024 IEEE Conference on Games, CoG 2024
    PublisherIEEE Computer Society
    ISBN (Electronic)9798350350678
    DOIs
    StatePublished - 2024
    Event6th Annual IEEE Conference on Games, CoG 2024 - Milan, Italy
    Duration: Aug 5 2024Aug 8 2024

    Publication series

    NameIEEE Conference on Computatonal Intelligence and Games, CIG
    ISSN (Print)2325-4270
    ISSN (Electronic)2325-4289

    Conference

    Conference6th Annual IEEE Conference on Games, CoG 2024
    Country/TerritoryItaly
    CityMilan
    Period8/5/248/8/24

    Keywords

    • procedural content generation
    • reinforcement learning

    ASJC Scopus subject areas

    • Artificial Intelligence
    • Computer Graphics and Computer-Aided Design
    • Computer Vision and Pattern Recognition
    • Human-Computer Interaction
    • Software

    Fingerprint

    Dive into the research topics of 'Scaling, Control and Generalization in Reinforcement Learning Level Generators'. Together they form a unique fingerprint.

    Cite this