Toward Mobile Mixed-Reality Interaction With Multi-Robot Systems

Jared Alan Frank, Sai Prasanth Krishnamoorthy, Vikram Kapila

Research output: Contribution to journalArticlepeer-review


Although human-multi-robot systems have received increased attention in recent years, current implementations rely on structured environments and utilize specialized, research-grade hardware to operate. This letter presents approaches that leverage the visual and inertial sensing of mobile devices to address the estimation and control challenges of multi-robot systems that function in shared spaces with human operators such that both the mobile device camera and robots can move freely in the environment. It is shown that a subset of robots in the system can be used to maintain a reference frame that facilitates tracking and control of the remaining robots to perform tasks, such as object retrieval, using an operator's mobile device as the only sensing and computational platform in the system. To evaluate the performance of the proposed approaches, experiments are conducted in which a system of mobile robots is commanded to retrieve objects in an environment. Results show that, compared to using the visual data alone, integrating both the visual and inertial data from mobile devices yields improvements in performance, flexibility, and computational efficiency in implementing human-multi-robot systems.

Original languageEnglish (US)
Article number7945250
Pages (from-to)1901-1908
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number4
StatePublished - Oct 2017


  • Distributed robot systems
  • and virtual reality and interfaces
  • telerobotics and teleoperation

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Biomedical Engineering
  • Human-Computer Interaction
  • Mechanical Engineering
  • Computer Vision and Pattern Recognition
  • Computer Science Applications
  • Control and Optimization
  • Artificial Intelligence


Dive into the research topics of 'Toward Mobile Mixed-Reality Interaction With Multi-Robot Systems'. Together they form a unique fingerprint.

Cite this