Holo-Dex: Teaching Dexterity with Immersive Mixed Reality

Sridhar Pandian Arunachalam, Irmak Guzey, Soumith Chintala, Lerrel Pinto

Research output: Chapter in Book/Report/Conference proceedingConference contribution


A fundamental challenge in teaching robots is to provide an effective interface for human teachers to demonstrate useful skills to a robot. This challenge is exacerbated in dexterous manipulation, where teaching high-dimensional, contact-rich behaviors often require esoteric teleoperation tools. In this work, we present Holo - Dex, a framework for dexter-ous manipulation that places a teacher in an immersive mixed reality through commodity VR headsets. The high-fidelity hand pose estimator onboard the headset is used to teleoperate the robot and collect demonstrations for a variety of general-purpose dexterous tasks. Given these demonstrations, we use powerful feature learning combined with non-parametric imi-tation to train dexterous skills. Our experiments on six common dexterous tasks, including in-hand rotation, spinning, and bottle opening, indicate that HOLO-DEX can both collect high-quality demonstration data and train skills in a matter of hours. Finally, we find that our trained skills can exhibit generalization on objects not seen in training. Videos of HOLO - DEX are available on {https://holo-dex.github.io/.}

Original languageEnglish (US)
Title of host publicationProceedings - ICRA 2023
Subtitle of host publicationIEEE International Conference on Robotics and Automation
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages8
ISBN (Electronic)9798350323658
StatePublished - 2023
Event2023 IEEE International Conference on Robotics and Automation, ICRA 2023 - London, United Kingdom
Duration: May 29 2023Jun 2 2023

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation
ISSN (Print)1050-4729


Conference2023 IEEE International Conference on Robotics and Automation, ICRA 2023
Country/TerritoryUnited Kingdom

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Electrical and Electronic Engineering
  • Artificial Intelligence


Dive into the research topics of 'Holo-Dex: Teaching Dexterity with Immersive Mixed Reality'. Together they form a unique fingerprint.

Cite this