Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning

Guy Davidson; Brenden M. Lake

Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning

Psychology

Research output: Contribution to conference › Paper › peer-review

Abstract

We explore the benefits of augmenting state-of-the-art model-free deep reinforcement learning with simple object representations. Following the Frostbite challenge posited by Lake et al. (2017), we identify object representations as a critical cognitive capacity lacking from current reinforcement learning agents. We discover that providing the Rainbow model (Hessel et al., 2018) with simple, feature-engineered object representations substantially boosts its performance on the Frostbite game from Atari 2600. We then analyze the relative contributions of the representations of different types of objects, identify environment states where these representations are most impactful, and examine how these representations aid in generalizing to novel situations.

Original language	English (US)
Pages	2023-2029
Number of pages	7
State	Published - 2020
Event	42nd Annual Meeting of the Cognitive Science Society: Developing a Mind: Learning in Humans, Animals, and Machines, CogSci 2020 - Virtual, Online Duration: Jul 29 2020 → Aug 1 2020

Conference

Conference	42nd Annual Meeting of the Cognitive Science Society: Developing a Mind: Learning in Humans, Animals, and Machines, CogSci 2020
City	Virtual, Online
Period	7/29/20 → 8/1/20

Keywords

deep reinforcement learning
DQN
model-free reinforcement learning
object representations

ASJC Scopus subject areas

Artificial Intelligence
Computer Science Applications
Human-Computer Interaction
Cognitive Neuroscience

Cite this

Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning. / Davidson, Guy; Lake, Brenden M.
2020. 2023-2029 Paper presented at 42nd Annual Meeting of the Cognitive Science Society: Developing a Mind: Learning in Humans, Animals, and Machines, CogSci 2020, Virtual, Online.

Research output: Contribution to conference › Paper › peer-review

@conference{05ce001fbc1f4aad88c7dda5d61cea29,

title = "Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning",

abstract = "We explore the benefits of augmenting state-of-the-art model-free deep reinforcement learning with simple object representations. Following the Frostbite challenge posited by Lake et al. (2017), we identify object representations as a critical cognitive capacity lacking from current reinforcement learning agents. We discover that providing the Rainbow model (Hessel et al., 2018) with simple, feature-engineered object representations substantially boosts its performance on the Frostbite game from Atari 2600. We then analyze the relative contributions of the representations of different types of objects, identify environment states where these representations are most impactful, and examine how these representations aid in generalizing to novel situations.",

keywords = "deep reinforcement learning, DQN, model-free reinforcement learning, object representations",

author = "Guy Davidson and Lake, {Brenden M.}",

note = "Funding Information: BML is grateful for discussions with Tomer Ullman, Josh Tenenbaum, and Sam Gershman while writing “Building machines that learn and think like people.” Those ideas sit at the heart of this paper. We also thank Reuben Feinman and Wai Keen Vong for helpful comments on this manuscript. Publisher Copyright: {\textcopyright} 2020 The Author(s); 42nd Annual Meeting of the Cognitive Science Society: Developing a Mind: Learning in Humans, Animals, and Machines, CogSci 2020 ; Conference date: 29-07-2020 Through 01-08-2020",

year = "2020",

language = "English (US)",

pages = "2023--2029",

}

TY - CONF

T1 - Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning

AU - Davidson, Guy

AU - Lake, Brenden M.

N1 - Funding Information: BML is grateful for discussions with Tomer Ullman, Josh Tenenbaum, and Sam Gershman while writing “Building machines that learn and think like people.” Those ideas sit at the heart of this paper. We also thank Reuben Feinman and Wai Keen Vong for helpful comments on this manuscript. Publisher Copyright: © 2020 The Author(s)

PY - 2020

Y1 - 2020

N2 - We explore the benefits of augmenting state-of-the-art model-free deep reinforcement learning with simple object representations. Following the Frostbite challenge posited by Lake et al. (2017), we identify object representations as a critical cognitive capacity lacking from current reinforcement learning agents. We discover that providing the Rainbow model (Hessel et al., 2018) with simple, feature-engineered object representations substantially boosts its performance on the Frostbite game from Atari 2600. We then analyze the relative contributions of the representations of different types of objects, identify environment states where these representations are most impactful, and examine how these representations aid in generalizing to novel situations.

AB - We explore the benefits of augmenting state-of-the-art model-free deep reinforcement learning with simple object representations. Following the Frostbite challenge posited by Lake et al. (2017), we identify object representations as a critical cognitive capacity lacking from current reinforcement learning agents. We discover that providing the Rainbow model (Hessel et al., 2018) with simple, feature-engineered object representations substantially boosts its performance on the Frostbite game from Atari 2600. We then analyze the relative contributions of the representations of different types of objects, identify environment states where these representations are most impactful, and examine how these representations aid in generalizing to novel situations.

KW - deep reinforcement learning

KW - DQN

KW - model-free reinforcement learning

KW - object representations

UR - http://www.scopus.com/inward/record.url?scp=85139497018&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85139497018&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:85139497018

SP - 2023

EP - 2029

T2 - 42nd Annual Meeting of the Cognitive Science Society: Developing a Mind: Learning in Humans, Animals, and Machines, CogSci 2020

Y2 - 29 July 2020 through 1 August 2020

ER -

Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning

Abstract

Conference

Keywords

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this