Mixed Reality Interface for Whole-Body Balancing and Manipulation of Humanoid Robot

The complexity of the control and operation is one of the roadblocks of widespread utilization of humanoid robots. In this study, we introduce a novel approach to humanoid robot control by leveraging a mixed reality (MR) interface for whole-body balancing and manipulation. This interface system uses an MR headset to track the operator's movement and provide the operator with useful visual information for the control. The robot mimics the operator's movement through a motion retargeting method based on linear scaling and inverse kinematics. The operator obtains visual access to the robot's perspective view augmented with fiducial detection and perceives the current stability of the robot by evaluating the robot's center-of-mass state in real-time against the precomputed balanced state basin. In experimental demonstrations, the operator successfully controlled the robot to grasp and lift an object without falling. The common issues in teleoperation with virtual reality headsets, motion sickness and unawareness of their surroundings, are reduced to a low level by using the MR headset with transparent glasses. This study demonstrates the potential of MR in teleoperation with a motion retargeting and stability monitoring method.