The Surprising Effectiveness of Representation Learning for Visual Imitation

Jyothish Pari, Nur Muhammad Mahi Shafiullah, Sridhar Pandian Arunachalam, Lerrel Pinto

Research output: Chapter in Book/Report/Conference proceedingConference contribution


While visual imitation learning offers one of the most effective ways of learning from visual demonstrations, generalizing from them requires either hundreds of diverse demonstrations, task specific priors, or large, hard-to-train parametric models. One reason such complexities arise is because standard visual imitation frameworks try to solve two coupled problems at once: learning a succinct but good representation from the diverse visual data, while simultaneously learning to associate the demonstrated actions with such representations. Such joint learning causes an interdependence between these two problems, which often results in needing large amounts of demonstrations for learning. To address this challenge, we instead propose to decouple representation learning from behavior learning for visual imitation. First, we learn a visual representation encoder from offline data using standard supervised and self-supervised learning methods. Once the representations are trained, we use non-parametric Locally Weighted Regression to predict the actions. We experimentally show that this simple decoupling improves the performance of visual imitation models on both offline demonstration datasets and real-robot door opening compared to prior work in visual imitation. All of our generated data, code, and robot videos are publicly available at

Original languageEnglish (US)
Title of host publicationRobotics
Subtitle of host publicationScience and Systems
EditorsKris Hauser, Dylan Shell, Shoudong Huang
PublisherMIT Press Journals
ISBN (Print)9780992374785
StatePublished - 2022
Event18th Robotics: Science and Systems, RSS 2022 - New York City, United States
Duration: Jun 27 2022 → …

Publication series

NameRobotics: Science and Systems
ISSN (Electronic)2330-765X


Conference18th Robotics: Science and Systems, RSS 2022
Country/TerritoryUnited States
CityNew York City
Period6/27/22 → …

ASJC Scopus subject areas

  • Artificial Intelligence
  • Control and Systems Engineering
  • Electrical and Electronic Engineering


Dive into the research topics of 'The Surprising Effectiveness of Representation Learning for Visual Imitation'. Together they form a unique fingerprint.

Cite this