Convolutional nets and watershed cuts for real-time semantic labeling of RGBD videos

Camille Couprie, Clément Farabet, Laurent Najman, Yann LeCun

Research output: Contribution to journalArticle

Abstract

This work addresses multi-class segmentation of indoor scenes with RGB-D inputs. While this area of research has gained much attention recently, most works still rely on handcrafted features. In contrast, we apply a multiscale convolutional network to learn features directly from the images and the depth information. Using a frame by frame labeling, we obtain nearly state-of-the-art performance on the NYU-v2 depth data set with an accuracy of 64.5%. We then show that the labeling can be further improved by exploiting the temporal consistency in the video sequence of the scene. To that goal, we present a method producing temporally consistent superpixels from a streaming video. Among the different methods producing superpixel segmentations of an image, the graph-based approach of Felzenszwalb and Huttenlocher is broadly employed. One of its interesting properties is that the regions are computed in a greedy manner in quasi-linear time by using a minimum spanning tree. In a framework exploiting minimum spanning trees all along, we propose an efficient video segmentation approach that computes temporally consistent pixels in a causal manner, filling the need for causal and real-time applications. We illustrate the labeling of indoor scenes in video sequences that could be processed in real-time using appropriate hardware such as an FPGA.

Original languageEnglish (US)
Article numberA20
Pages (from-to)3489-3511
Number of pages23
JournalJournal of Machine Learning Research
Volume15
StatePublished - Jan 1 2015

Keywords

  • Convolutional networks
  • Deep learning
  • Depth information
  • Optimization
  • Superpixels

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Convolutional nets and watershed cuts for real-time semantic labeling of RGBD videos'. Together they form a unique fingerprint.

  • Cite this