Playing Atari with few neurons: Improving the efficacy of reinforcement learning by decoupling feature extraction and decision making

Giuseppe Cuccu, Julian Togelius, Philippe Cudré-Mauroux

    Research output: Contribution to journalArticlepeer-review

    Abstract

    We propose a new method for learning compact state representations and policies separately but simultaneously for policy approximation in vision-based applications such as Atari games. Approaches based on deep reinforcement learning typically map pixels directly to actions to enable end-to-end training. Internally, however, the deep neural network bears the responsibility of both extracting useful information and making decisions based on it, two objectives which can be addressed independently. Separating the image processing from the action selection allows for a better understanding of either task individually, as well as potentially finding smaller policy representations which is inherently interesting. Our approach learns state representations using a compact encoder based on two novel algorithms: (i) Increasing Dictionary Vector Quantization builds a dictionary of state representations which grows in size over time, allowing our method to address new observations as they appear in an open-ended online-learning context; and (ii) Direct Residuals Sparse Coding encodes observations in function of the dictionary, aiming for highest information inclusion by disregarding reconstruction error and maximizing code sparsity. As the dictionary size increases, however, the encoder produces increasingly larger inputs for the neural network; this issue is addressed with a new variant of the Exponential Natural Evolution Strategies algorithm which adapts the dimensionality of its probability distribution along the run. We test our system on a selection of Atari games using tiny neural networks of only 6 to 18 neurons (depending on each game’s controls). These are still capable of achieving results that are not much worse, and occasionally superior, to the state-of-the-art in direct policy search which uses two orders of magnitude more neurons.

    Original languageEnglish (US)
    Article number17
    JournalAutonomous Agents and Multi-Agent Systems
    Volume35
    Issue number2
    DOIs
    StatePublished - Oct 2021

    Keywords

    • Evolutionary algorithms
    • Game playing
    • Learning agent capabilities
    • Neuroevolution

    ASJC Scopus subject areas

    • Artificial Intelligence

    Fingerprint Dive into the research topics of 'Playing Atari with few neurons: Improving the efficacy of reinforcement learning by decoupling feature extraction and decision making'. Together they form a unique fingerprint.

    Cite this