Through-wall object recognition and pose estimation

Ruoyu Wang, Siyuan Xiang, Chen Feng, Pu Wang, Semiha Ergan, Yi Fang

Research output: Contribution to conferencePaperpeer-review


Robots need to perceive beyond lines of sight, e.g., to avoid cutting water pipes or electric wires when drilling holes on a wall. Recent off-the-shelf radio frequency (RF) imaging sensors ease the process of 3D sensing inside or through walls. Yet unlike optical images, RF images are difficult to understand by a human. Meanwhile, in practice, RF components are often subject to hardware imperfections, resulting in distorted RF images, whose quality could be far from the claimed specifications. Thus, we introduce several challenging geometric and semantic perception tasks on such signals, including object and material recognition, fine-grained property classification and pose estimation. Since detailed forward modeling of such sensors is sometimes difficult, due to hidden or inaccessible system parameters, onboard processing procedures and limited access to raw RF waveform, we tackled the above tasks by supervised machine learning. We collected a large dataset of RF images of utility objects through a mock wall as the input of our algorithm, and the corresponding optical images were taken from the other side of the wall simultaneously as the ground truth. We designed three learning algorithms based on nearest neighbors or neural networks, and report their performances on the dataset. Our experiments showed reasonable results for semantic perception tasks yet unsatisfactory results for geometric ones, calling for more efforts in this research direction.

Original languageEnglish (US)
Number of pages8
StatePublished - 2019
Event36th International Symposium on Automation and Robotics in Construction, ISARC 2019 - Banff, Canada
Duration: May 21 2019May 24 2019


Conference36th International Symposium on Automation and Robotics in Construction, ISARC 2019


  • Deep learning
  • Object recognition
  • Pose estimation
  • Through-wall imaging

ASJC Scopus subject areas

  • Artificial Intelligence
  • Building and Construction
  • Human-Computer Interaction


Dive into the research topics of 'Through-wall object recognition and pose estimation'. Together they form a unique fingerprint.

Cite this