Understanding 3D Object Interaction from a Single Image

Shengyi Qian; David F. Fouhey

doi:10.1109/ICCV51070.2023.01988

Understanding 3D Object Interaction from a Single Image

Shengyi Qian, David F. Fouhey

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Humans can easily understand a single image as depicting multiple potential objects permitting interaction. We use this skill to plan our interactions with the world and accelerate understanding new objects without engaging in interaction. In this paper, we would like to endow machines with the similar ability, so that intelligent agents can better explore the 3D scene or manipulate objects. Our approach is a transformer-based model that predicts the 3D location, physical properties and affordance of objects. To power this model, we collect a dataset with Internet videos, egocentric videos and indoor images to train and validate our approach. Our model yields strong performance on our data, and generalizes well to robotics data.

Original language	English (US)
Title of host publication	Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	21696-21706
Number of pages	11
ISBN (Electronic)	9798350307184
DOIs	https://doi.org/10.1109/ICCV51070.2023.01988
State	Published - 2023
Event	2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023 - Paris, France Duration: Oct 2 2023 → Oct 6 2023

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
ISSN (Print)	1550-5499

Conference

Conference	2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023
Country/Territory	France
City	Paris
Period	10/2/23 → 10/6/23

ASJC Scopus subject areas

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/ICCV51070.2023.01988

Cite this

Understanding 3D Object Interaction from a Single Image. / Qian, Shengyi; Fouhey, David F.
Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 21696-21706 (Proceedings of the IEEE International Conference on Computer Vision).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Qian, S & Fouhey, DF 2023, Understanding 3D Object Interaction from a Single Image. in Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023. Proceedings of the IEEE International Conference on Computer Vision, Institute of Electrical and Electronics Engineers Inc., pp. 21696-21706, 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, 10/2/23. https://doi.org/10.1109/ICCV51070.2023.01988

@inproceedings{f429b7604ca348ac8711aa7214cd7a12,

title = "Understanding 3D Object Interaction from a Single Image",

abstract = "Humans can easily understand a single image as depicting multiple potential objects permitting interaction. We use this skill to plan our interactions with the world and accelerate understanding new objects without engaging in interaction. In this paper, we would like to endow machines with the similar ability, so that intelligent agents can better explore the 3D scene or manipulate objects. Our approach is a transformer-based model that predicts the 3D location, physical properties and affordance of objects. To power this model, we collect a dataset with Internet videos, egocentric videos and indoor images to train and validate our approach. Our model yields strong performance on our data, and generalizes well to robotics data.",

author = "Shengyi Qian and Fouhey, {David F.}",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023 ; Conference date: 02-10-2023 Through 06-10-2023",

year = "2023",

doi = "10.1109/ICCV51070.2023.01988",

language = "English (US)",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "21696--21706",

booktitle = "Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023",

}

TY - GEN

T1 - Understanding 3D Object Interaction from a Single Image

AU - Qian, Shengyi

AU - Fouhey, David F.

PY - 2023

Y1 - 2023

N2 - Humans can easily understand a single image as depicting multiple potential objects permitting interaction. We use this skill to plan our interactions with the world and accelerate understanding new objects without engaging in interaction. In this paper, we would like to endow machines with the similar ability, so that intelligent agents can better explore the 3D scene or manipulate objects. Our approach is a transformer-based model that predicts the 3D location, physical properties and affordance of objects. To power this model, we collect a dataset with Internet videos, egocentric videos and indoor images to train and validate our approach. Our model yields strong performance on our data, and generalizes well to robotics data.

AB - Humans can easily understand a single image as depicting multiple potential objects permitting interaction. We use this skill to plan our interactions with the world and accelerate understanding new objects without engaging in interaction. In this paper, we would like to endow machines with the similar ability, so that intelligent agents can better explore the 3D scene or manipulate objects. Our approach is a transformer-based model that predicts the 3D location, physical properties and affordance of objects. To power this model, we collect a dataset with Internet videos, egocentric videos and indoor images to train and validate our approach. Our model yields strong performance on our data, and generalizes well to robotics data.

UR - http://www.scopus.com/inward/record.url?scp=85183449667&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85183449667&partnerID=8YFLogxK

U2 - 10.1109/ICCV51070.2023.01988

DO - 10.1109/ICCV51070.2023.01988

M3 - Conference contribution

AN - SCOPUS:85183449667

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 21696

EP - 21706

BT - Proceedings - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 IEEE/CVF International Conference on Computer Vision, ICCV 2023

Y2 - 2 October 2023 through 6 October 2023

ER -

Understanding 3D Object Interaction from a Single Image

Abstract

Publication series

Conference

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this