A multimodal intention detection sensor suite for shared autonomy of upper-limb robotic prostheses

Marcus Gardner, C. Sebastian Mancero Castillo, Samuel Wilson, Dario Farina, Etienne Burdet, Boo Cheong Khoo, S. Farokh Atashzar, Ravi Vaidyanathan

Research output: Contribution to journalArticlepeer-review


Neurorobotic augmentation (e.g., robotic assist) is now in regular use to support individuals suffering from impaired motor functions. A major unresolved challenge, however, is the excessive cognitive load necessary for the human-machine interface (HMI). Grasp control remains one of the most challenging HMI tasks, demanding simultaneous, agile, and precise control of multiple degrees-of-freedom (DoFs) while following a specific timing pattern in the joint and human-robot task spaces. Most commercially available systems use either an indirect mode-switching configuration or a limited sequential control strategy, limiting activation to one DoF at a time. To address this challenge, we introduce a shared autonomy framework centred around a low-cost multi-modal sensor suite fusing: (a) mechanomyography (MMG) to estimate the intended muscle activation, (b) camera-based visual information for integrated autonomous object recognition, and (c) inertial measurement to enhance intention prediction based on the grasping trajectory. The complete system predicts user intent for grasp based on measured dynamical features during natural motions. A total of 84 motion features were extracted from the sensor suite, and tests were conducted on 10 able-bodied and 1 amputee participants for grasping common household objects with a robotic hand. Real-time grasp classification accuracy using visual and motion features obtained 100%, 82.5%, and 88.9% across all participants for detecting and executing grasping actions for a bottle, lid, and box, respectively. The proposed multimodal sensor suite is a novel approach for predicting different grasp strategies and automating task performance using a commercial upper-limb prosthetic device. The system also shows potential to improve the usability of modern neurorobotic systems due to the intuitive control design.

Original languageEnglish (US)
Article number6097
Pages (from-to)1-20
Number of pages20
JournalSensors (Switzerland)
Issue number21
StatePublished - Nov 1 2020


  • Mechanomyography
  • Prosthetic technology
  • Shared autonomy
  • Humans
  • Prostheses and Implants
  • Intention
  • Upper Extremity
  • Electromyography
  • Hand
  • Robotics

ASJC Scopus subject areas

  • Analytical Chemistry
  • Information Systems
  • Instrumentation
  • Atomic and Molecular Physics, and Optics
  • Electrical and Electronic Engineering
  • Biochemistry


Dive into the research topics of 'A multimodal intention detection sensor suite for shared autonomy of upper-limb robotic prostheses'. Together they form a unique fingerprint.

Cite this