Exploring Adversarial Robustness of Multi-sensor Perception Systems in Self Driving

James Tu, Huichen Li, Xinchen Yan, Mengye Ren, Yun Chen, Ming Liang, Eilyan Bitar, Ersin Yumer, Raquel Urtasun

Research output: Contribution to journalConference articlepeer-review


Modern self-driving perception systems have been shown to improve upon processing complementary inputs such as LiDAR with images. In isolation, 2D images have been found to be extremely vulnerable to adversarial attacks. Yet, there are limited studies on the adversarial robustness of multi-modal models that fuse LiDAR and image features. Furthermore, existing works do not consider physically realizable perturbations that are consistent across the input modalities. In this paper, we showcase practical susceptibilities of multi-sensor detection by inserting an adversarial object on a host vehicle. We focus on physically realizable and input-agnostic attacks that are feasible to execute in practice, and show that a single universal adversary can hide different host vehicles from state-of-the-art multi-modal detectors. Our experiments demonstrate that successful attacks are primarily caused by easily corrupted image features. Furthermore, in modern sensor fusion methods which project image features into 3D, adversarial attacks can exploit the projection process to generate false positives in distant regions in 3D. Towards more robust multi-modal perception systems, we show that adversarial training with feature denoising can boost robustness to such attacks significantly.

Original languageEnglish (US)
Pages (from-to)1013-1024
Number of pages12
JournalProceedings of Machine Learning Research
StatePublished - 2021
Event5th Conference on Robot Learning, CoRL 2021 - London, United Kingdom
Duration: Nov 8 2021Nov 11 2021


  • Adversarial
  • Multimodal
  • Perception
  • Self-Driving

ASJC Scopus subject areas

  • Artificial Intelligence
  • Software
  • Control and Systems Engineering
  • Statistics and Probability


Dive into the research topics of 'Exploring Adversarial Robustness of Multi-sensor Perception Systems in Self Driving'. Together they form a unique fingerprint.

Cite this