TY - JOUR
T1 - ARGUS
T2 - Visualization of AI-Assisted Task Guidance in AR
AU - Castelo, Sonia
AU - Rulff, Joao
AU - McGowan, Erin
AU - Steers, Bea
AU - Wu, Guande
AU - Chen, Shaoyu
AU - Roman, Iran
AU - Lopez, Roque
AU - Brewer, Ethan
AU - Zhao, Chen
AU - Qian, Jing
AU - Cho, Kyunghyun
AU - He, He
AU - Sun, Qi
AU - Vo, Huy
AU - Bello, Juan
AU - Krone, Michael
AU - Silva, Claudio
N1 - Publisher Copyright:
© 1995-2012 IEEE.
PY - 2024/1/1
Y1 - 2024/1/1
N2 - The concept of augmented reality (AR) assistants has captured the human imagination for decades, becoming a staple of modern science fiction. To pursue this goal, it is necessary to develop artificial intelligence (AI)-based methods that simultaneously perceive the 3D environment, reason about physical tasks, and model the performer, all in real-time. Within this framework, a wide variety of sensors are needed to generate data across different modalities, such as audio, video, depth, speech, and time-of-flight. The required sensors are typically part of the AR headset, providing performer sensing and interaction through visual, audio, and haptic feedback. AI assistants not only record the performer as they perform activities, but also require machine learning (ML) models to understand and assist the performer as they interact with the physical world. Therefore, developing such assistants is a challenging task. We propose ARGUS, a visual analytics system to support the development of intelligent AR assistants. Our system was designed as part of a multi-year-long collaboration between visualization researchers and ML and AR experts. This co-design process has led to advances in the visualization of ML in AR. Our system allows for online visualization of object, action, and step detection as well as offline analysis of previously recorded AR sessions. It visualizes not only the multimodal sensor data streams but also the output of the ML models. This allows developers to gain insights into the performer activities as well as the ML models, helping them troubleshoot, improve, and fine-tune the components of the AR assistant.
AB - The concept of augmented reality (AR) assistants has captured the human imagination for decades, becoming a staple of modern science fiction. To pursue this goal, it is necessary to develop artificial intelligence (AI)-based methods that simultaneously perceive the 3D environment, reason about physical tasks, and model the performer, all in real-time. Within this framework, a wide variety of sensors are needed to generate data across different modalities, such as audio, video, depth, speech, and time-of-flight. The required sensors are typically part of the AR headset, providing performer sensing and interaction through visual, audio, and haptic feedback. AI assistants not only record the performer as they perform activities, but also require machine learning (ML) models to understand and assist the performer as they interact with the physical world. Therefore, developing such assistants is a challenging task. We propose ARGUS, a visual analytics system to support the development of intelligent AR assistants. Our system was designed as part of a multi-year-long collaboration between visualization researchers and ML and AR experts. This co-design process has led to advances in the visualization of ML in AR. Our system allows for online visualization of object, action, and step detection as well as offline analysis of previously recorded AR sessions. It visualizes not only the multimodal sensor data streams but also the output of the ML models. This allows developers to gain insights into the performer activities as well as the ML models, helping them troubleshoot, improve, and fine-tune the components of the AR assistant.
KW - AR/VR/Immersive
KW - Application Motivated Visualization
KW - Data Models
KW - Image and Video Data
KW - Temporal Data
UR - http://www.scopus.com/inward/record.url?scp=85181560614&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85181560614&partnerID=8YFLogxK
U2 - 10.1109/TVCG.2023.3327396
DO - 10.1109/TVCG.2023.3327396
M3 - Article
AN - SCOPUS:85181560614
SN - 1077-2626
VL - 30
SP - 1313
EP - 1323
JO - IEEE Transactions on Visualization and Computer Graphics
JF - IEEE Transactions on Visualization and Computer Graphics
IS - 1
ER -