PlaneFormers: From Sparse View Planes to 3D Reconstruction

Samir Agarwala; Linyi Jin; Chris Rockwell; David F. Fouhey

doi:10.1007/978-3-031-20062-5_12

PlaneFormers: From Sparse View Planes to 3D Reconstruction

Samir Agarwala, Linyi Jin, Chris Rockwell, David F. Fouhey

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present an approach for the planar surface reconstruction of a scene from images with limited overlap. This reconstruction task is challenging since it requires jointly reasoning about single image 3D reconstruction, correspondence between images, and the relative camera pose between images. Past work has proposed optimization-based approaches. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning. Our experiments show that our approach is substantially more effective than prior work, and that several 3D-specific design decisions are crucial for its success. Code is available at https://github.com/samiragarwala/PlaneFormers.

Original language	English (US)
Title of host publication	Computer Vision – ECCV 2022 - 17th European Conference, Proceedings
Editors	Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	192-209
Number of pages	18
ISBN (Print)	9783031200618
DOIs	https://doi.org/10.1007/978-3-031-20062-5_12
State	Published - 2022
Event	17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel Duration: Oct 23 2022 → Oct 27 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13663 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th European Conference on Computer Vision, ECCV 2022
Country/Territory	Israel
City	Tel Aviv
Period	10/23/22 → 10/27/22

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-20062-5_12

Cite this

Agarwala, S., Jin, L., Rockwell, C., & Fouhey, D. F. (2022). PlaneFormers: From Sparse View Planes to 3D Reconstruction. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings (pp. 192-209). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13663 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-20062-5_12

PlaneFormers: From Sparse View Planes to 3D Reconstruction. / Agarwala, Samir; Jin, Linyi; Rockwell, Chris et al.
Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. ed. / Shai Avidan; Gabriel Brostow; Moustapha Cissé; Giovanni Maria Farinella; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. p. 192-209 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13663 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Agarwala, S, Jin, L, Rockwell, C & Fouhey, DF 2022, PlaneFormers: From Sparse View Planes to 3D Reconstruction. in S Avidan, G Brostow, M Cissé, GM Farinella & T Hassner (eds), Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13663 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 192-209, 17th European Conference on Computer Vision, ECCV 2022, Tel Aviv, Israel, 10/23/22. https://doi.org/10.1007/978-3-031-20062-5_12

Agarwala S, Jin L, Rockwell C, Fouhey DF. PlaneFormers: From Sparse View Planes to 3D Reconstruction. In Avidan S, Brostow G, Cissé M, Farinella GM, Hassner T, editors, Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 192-209. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-20062-5_12

Agarwala, Samir ; Jin, Linyi ; Rockwell, Chris et al. / PlaneFormers : From Sparse View Planes to 3D Reconstruction. Computer Vision – ECCV 2022 - 17th European Conference, Proceedings. editor / Shai Avidan ; Gabriel Brostow ; Moustapha Cissé ; Giovanni Maria Farinella ; Tal Hassner. Springer Science and Business Media Deutschland GmbH, 2022. pp. 192-209 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{73a0934f7ee041efb3443dabdad87de8,

title = "PlaneFormers: From Sparse View Planes to 3D Reconstruction",

abstract = "We present an approach for the planar surface reconstruction of a scene from images with limited overlap. This reconstruction task is challenging since it requires jointly reasoning about single image 3D reconstruction, correspondence between images, and the relative camera pose between images. Past work has proposed optimization-based approaches. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning. Our experiments show that our approach is substantially more effective than prior work, and that several 3D-specific design decisions are crucial for its success. Code is available at https://github.com/samiragarwala/PlaneFormers.",

author = "Samir Agarwala and Linyi Jin and Chris Rockwell and Fouhey, {David F.}",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 17th European Conference on Computer Vision, ECCV 2022 ; Conference date: 23-10-2022 Through 27-10-2022",

year = "2022",

doi = "10.1007/978-3-031-20062-5_12",

language = "English (US)",

isbn = "9783031200618",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "192--209",

editor = "Shai Avidan and Gabriel Brostow and Moustapha Ciss{\'e} and Farinella, {Giovanni Maria} and Tal Hassner",

booktitle = "Computer Vision – ECCV 2022 - 17th European Conference, Proceedings",

address = "Germany",

}

TY - GEN

T1 - PlaneFormers

T2 - 17th European Conference on Computer Vision, ECCV 2022

AU - Agarwala, Samir

AU - Jin, Linyi

AU - Rockwell, Chris

AU - Fouhey, David F.

PY - 2022

Y1 - 2022

N2 - We present an approach for the planar surface reconstruction of a scene from images with limited overlap. This reconstruction task is challenging since it requires jointly reasoning about single image 3D reconstruction, correspondence between images, and the relative camera pose between images. Past work has proposed optimization-based approaches. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning. Our experiments show that our approach is substantially more effective than prior work, and that several 3D-specific design decisions are crucial for its success. Code is available at https://github.com/samiragarwala/PlaneFormers.

AB - We present an approach for the planar surface reconstruction of a scene from images with limited overlap. This reconstruction task is challenging since it requires jointly reasoning about single image 3D reconstruction, correspondence between images, and the relative camera pose between images. Past work has proposed optimization-based approaches. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning. Our experiments show that our approach is substantially more effective than prior work, and that several 3D-specific design decisions are crucial for its success. Code is available at https://github.com/samiragarwala/PlaneFormers.

UR - http://www.scopus.com/inward/record.url?scp=85144504531&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85144504531&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-20062-5_12

DO - 10.1007/978-3-031-20062-5_12

M3 - Conference contribution

AN - SCOPUS:85144504531

SN - 9783031200618

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 192

EP - 209

BT - Computer Vision – ECCV 2022 - 17th European Conference, Proceedings

A2 - Avidan, Shai

A2 - Brostow, Gabriel

A2 - Cissé, Moustapha

A2 - Farinella, Giovanni Maria

A2 - Hassner, Tal

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 23 October 2022 through 27 October 2022

ER -