Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality

Jianzhe Lin; Shaoyu Chen; Bea Steers; Huy T. Vo; Claudio T. Silva; Qi Sun

doi:10.1109/VRW58643.2023.00236

Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality

Jianzhe Lin, Shaoyu Chen, Bea Steers, Huy T. Vo, Claudio T. Silva, Qi Sun

Urban Initiative

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Augmented Reality (AR) devices are equipped with advanced sensors such as panoramic cameras. They can see or hear what humans cannot. The superior sensing capabilities - together with machine perception models - enable promising AR-based human assistants in daily tasks. Human task performance progressively enhances with accumulated memory, such as with observed objects' location in working memory, and training in long-term memory. We transform this cognitive nature into real-time machine perception models tailored for human-assisting AR applications. Instead of running machine perception models on each consecutive frame, we leverage the egocentric AR cameras to reconstruct a '4D memory map', which mimics the human spatiotemporal memory.

Original language	English (US)
Title of host publication	Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	787-788
Number of pages	2
ISBN (Electronic)	9798350348392
DOIs	https://doi.org/10.1109/VRW58643.2023.00236
State	Published - 2023
Event	2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023 - Shanghai, China Duration: Mar 25 2023 → Mar 29 2023

Publication series

Name	Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

Conference

Conference	2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023
Country/Territory	China
City	Shanghai
Period	3/25/23 → 3/29/23

Keywords

Modeling and Simulation
Perception and cognition
cognitive nature
memory
real time machine perception

ASJC Scopus subject areas

Computer Graphics and Computer-Aided Design
Computer Science Applications
Human-Computer Interaction
Media Technology

Access to Document

10.1109/VRW58643.2023.00236

Cite this

Lin, J., Chen, S., Steers, B., Vo, H. T., Silva, C. T., & Sun, Q. (2023). Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality. In Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023 (pp. 787-788). (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/VRW58643.2023.00236

Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality. / Lin, Jianzhe; Chen, Shaoyu; Steers, Bea et al.
Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. Institute of Electrical and Electronics Engineers Inc., 2023. p. 787-788 (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lin, J, Chen, S, Steers, B, Vo, HT, Silva, CT & Sun, Q 2023, Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality. in Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023, Institute of Electrical and Electronics Engineers Inc., pp. 787-788, 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023, Shanghai, China, 3/25/23. https://doi.org/10.1109/VRW58643.2023.00236

Lin J, Chen S, Steers B, Vo HT, Silva CT, Sun Q. Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality. In Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. Institute of Electrical and Electronics Engineers Inc. 2023. p. 787-788. (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023). doi: 10.1109/VRW58643.2023.00236

Lin, Jianzhe ; Chen, Shaoyu ; Steers, Bea et al. / Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality. Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023. Institute of Electrical and Electronics Engineers Inc., 2023. pp. 787-788 (Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023).

@inproceedings{84d91d247e8c4b86b71e590c18fd84cc,

title = "Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality",

abstract = "Augmented Reality (AR) devices are equipped with advanced sensors such as panoramic cameras. They can see or hear what humans cannot. The superior sensing capabilities - together with machine perception models - enable promising AR-based human assistants in daily tasks. Human task performance progressively enhances with accumulated memory, such as with observed objects' location in working memory, and training in long-term memory. We transform this cognitive nature into real-time machine perception models tailored for human-assisting AR applications. Instead of running machine perception models on each consecutive frame, we leverage the egocentric AR cameras to reconstruct a '4D memory map', which mimics the human spatiotemporal memory.",

keywords = "Modeling and Simulation, Perception and cognition, cognitive nature, memory, real time machine perception",

author = "Jianzhe Lin and Shaoyu Chen and Bea Steers and Vo, {Huy T.} and Silva, {Claudio T.} and Qi Sun",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023 ; Conference date: 25-03-2023 Through 29-03-2023",

year = "2023",

doi = "10.1109/VRW58643.2023.00236",

language = "English (US)",

series = "Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "787--788",

booktitle = "Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023",

}

TY - GEN

T1 - Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality

AU - Lin, Jianzhe

AU - Chen, Shaoyu

AU - Steers, Bea

AU - Vo, Huy T.

AU - Silva, Claudio T.

AU - Sun, Qi

PY - 2023

Y1 - 2023

N2 - Augmented Reality (AR) devices are equipped with advanced sensors such as panoramic cameras. They can see or hear what humans cannot. The superior sensing capabilities - together with machine perception models - enable promising AR-based human assistants in daily tasks. Human task performance progressively enhances with accumulated memory, such as with observed objects' location in working memory, and training in long-term memory. We transform this cognitive nature into real-time machine perception models tailored for human-assisting AR applications. Instead of running machine perception models on each consecutive frame, we leverage the egocentric AR cameras to reconstruct a '4D memory map', which mimics the human spatiotemporal memory.

AB - Augmented Reality (AR) devices are equipped with advanced sensors such as panoramic cameras. They can see or hear what humans cannot. The superior sensing capabilities - together with machine perception models - enable promising AR-based human assistants in daily tasks. Human task performance progressively enhances with accumulated memory, such as with observed objects' location in working memory, and training in long-term memory. We transform this cognitive nature into real-time machine perception models tailored for human-assisting AR applications. Instead of running machine perception models on each consecutive frame, we leverage the egocentric AR cameras to reconstruct a '4D memory map', which mimics the human spatiotemporal memory.

KW - Modeling and Simulation

KW - Perception and cognition

KW - cognitive nature

KW - memory

KW - real time machine perception

UR - http://www.scopus.com/inward/record.url?scp=85159700013&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85159700013&partnerID=8YFLogxK

U2 - 10.1109/VRW58643.2023.00236

DO - 10.1109/VRW58643.2023.00236

M3 - Conference contribution

AN - SCOPUS:85159700013

T3 - Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

SP - 787

EP - 788

BT - Proceedings - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops, VRW 2023

Y2 - 25 March 2023 through 29 March 2023

ER -

Spatiotemporal-Memory-Guided Machine Perception for Augmented Reality

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this