Stereo matching by training a convolutional neural network to compare image patches

Jure Žbontar, Yann Lecun

Research output: Contribution to journalArticlepeer-review


We present a method for extracting depth information from a rectified image pair. Our approach focuses on the first stage of many stereo algorithms: the matching cost computation. We approach the problem by learning a similarity measure on small image patches using a convolutional neural network. Training is carried out in a supervised manner by constructing a binary classification data set with examples of similar and dissimilar pairs of patches. We examine two network architectures for this task: one tuned for speed, the other for accuracy. The output of the convolutional neural network is used to initialize the stereo matching cost. A series of post-processing steps follow: cross-based cost aggregation, semiglobal matching, a left-right consistency check, subpixel enhancement, a median filter, and a bilateral filter. We evaluate our method on the KITTI 2012, KITTI 2015, and Middlebury stereo data sets and show that it outperforms other approaches on all three data sets.

Original languageEnglish (US)
JournalJournal of Machine Learning Research
StatePublished - Apr 1 2016


  • Convolutional neural networks
  • Matching cost
  • Similarity learning
  • Stereo
  • Supervised learning

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Statistics and Probability
  • Artificial Intelligence


Dive into the research topics of 'Stereo matching by training a convolutional neural network to compare image patches'. Together they form a unique fingerprint.

Cite this