TY - GEN
T1 - Directed Ray Distance Functions for 3D Scene Reconstruction
AU - Kulkarni, Nilesh
AU - Johnson, Justin
AU - Fouhey, David F.
N1 - Funding Information:
Acknowledgement. We would like the thank Alexandar Raistrick and Chris Rockwell for their help with the 3DFront dataset. We like to thank Shubham Tulsiani, Ekdeep Singh Lubana, Richard Higgins, Sarah Jabour, Shengyi Qian, Linyi Jin, Karan Desai, Mohammed El Banani, Chris Rockwell, Alexandar Raistrick, Dandan Shan, Andrew Owens for comments on the draft versions of this paper. NK was supported by TRI. Toyota Research Institute (“TRI”) provided funds to assist the authors with their research but this article solely reflects the opinions and conclusions of its authors and not TRI or any other Toyota entity
Publisher Copyright:
© 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.
PY - 2022
Y1 - 2022
N2 - We present an approach for full 3D scene reconstruction from a single unseen image. We trained on dataset of realistic non-watertight scans of scenes. Our approach uses a predicted distance function, since these have shown promise in handling complex topologies and large spaces. We identify and analyze two key challenges for predicting such image conditioned distance functions that have prevented their success on real 3D scene data. First, we show that predicting a conventional scene distance from an image requires reasoning over a large receptive field. Second, we analytically show that the optimal output of the network trained to predict these distance functions does not obey all the distance function properties. We propose an alternate distance function, the Directed Ray Distance Function (DRDF), that tackles both challenges. We show that a deep network trained to predict DRDFs outperforms all other methods quantitatively and qualitatively on 3D reconstruction from single image on Matterport3D, 3DFront, and ScanNet. (Project Page: https://nileshkulkarni.github.io/scene_drdf
AB - We present an approach for full 3D scene reconstruction from a single unseen image. We trained on dataset of realistic non-watertight scans of scenes. Our approach uses a predicted distance function, since these have shown promise in handling complex topologies and large spaces. We identify and analyze two key challenges for predicting such image conditioned distance functions that have prevented their success on real 3D scene data. First, we show that predicting a conventional scene distance from an image requires reasoning over a large receptive field. Second, we analytically show that the optimal output of the network trained to predict these distance functions does not obey all the distance function properties. We propose an alternate distance function, the Directed Ray Distance Function (DRDF), that tackles both challenges. We show that a deep network trained to predict DRDFs outperforms all other methods quantitatively and qualitatively on 3D reconstruction from single image on Matterport3D, 3DFront, and ScanNet. (Project Page: https://nileshkulkarni.github.io/scene_drdf
KW - Distance functions
KW - Single image 3D
UR - http://www.scopus.com/inward/record.url?scp=85142720393&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85142720393&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-20086-1_12
DO - 10.1007/978-3-031-20086-1_12
M3 - Conference contribution
AN - SCOPUS:85142720393
SN - 9783031200854
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 201
EP - 219
BT - Computer Vision – ECCV 2022 - 17th European Conference, Proceedings
A2 - Avidan, Shai
A2 - Brostow, Gabriel
A2 - Cissé, Moustapha
A2 - Farinella, Giovanni Maria
A2 - Hassner, Tal
PB - Springer Science and Business Media Deutschland GmbH
T2 - 17th European Conference on Computer Vision, ECCV 2022
Y2 - 23 October 2022 through 27 October 2022
ER -