TY - GEN
T1 - Personalized Productive Engagement Recognition in Robot-Mediated Collaborative Learning
AU - Chithrra Raghuram, Vetha Vikashini
AU - Salam, Hanan
AU - Nasir, Jauwairia
AU - Bruno, Barbara
AU - Celiktutan, Oya
N1 - Funding Information:
This research was supported by NYUAD internal grant and by the Center of AI & Robotics (CAIR) grant. The work of Jauwairia Nasir has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 765955. The work of Oya Celiktutan was supported by the LISI project, funded by the UKRI EPSRC (Grant Ref.: EP/V010875/1).
Funding Information:
This research was supported by NYUAD internal grant and by the Center of AI & Robotics (CAIR) grant. The work of Jauwairia Nasir has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 765955. The work of Oya Celiktutan was supported by the LISI project, funded by the UKRI EPSRC (Grant Ref.: EP/V010875/1).
Publisher Copyright:
© 2022 ACM.
PY - 2022/11/7
Y1 - 2022/11/7
N2 - In this paper, we propose and compare personalized models for Productive Engagement (PE) recognition. PE is defined as the level of engagement that maximizes learning. Previously, in the context of robot-mediated collaborative learning, a framework of productive engagement was developed by utilizing multimodal data of 32 dyads and learning profiles, namely, Expressive Explorers (EE), Calm Tinkerers (CT), and Silent Wanderers (SW) were identified which categorize learners according to their learning gain. Within the same framework, a PE score was constructed in a non-supervised manner for real-time evaluation. Here, we use these profiles and the PE score within an AutoML deep learning framework to personalize PE models. We investigate two approaches for this purpose: (1) Single-task Deep Neural Architecture Search (ST-NAS), and (2) Multitask NAS (MT-NAS). In the former approach, personalized models for each learner profile are learned from multimodal features and compared to non-personalized models. In the MT-NAS approach, we investigate whether jointly classifying the learners' profiles with the engagement score through multi-task learning would serve as an implicit personalization of PE. Moreover, we compare the predictive power of two types of features: incremental and non-incremental features. Non-incremental features correspond to features computed from the participant's behaviours in fixed time windows. Incremental features are computed by accounting to the behaviour from the beginning of the learning activity till the time window where productive engagement is observed. Our experimental results show that (1) personalized models improve the recognition performance with respect to non-personalized models when training models for the gainer vs. non-gainer groups, (2) multitask NAS (implicit personalization) also outperforms non-personalized models, (3) the speech modality has high contribution towards prediction, and (4) non-incremental features outperform the incremental ones overall.
AB - In this paper, we propose and compare personalized models for Productive Engagement (PE) recognition. PE is defined as the level of engagement that maximizes learning. Previously, in the context of robot-mediated collaborative learning, a framework of productive engagement was developed by utilizing multimodal data of 32 dyads and learning profiles, namely, Expressive Explorers (EE), Calm Tinkerers (CT), and Silent Wanderers (SW) were identified which categorize learners according to their learning gain. Within the same framework, a PE score was constructed in a non-supervised manner for real-time evaluation. Here, we use these profiles and the PE score within an AutoML deep learning framework to personalize PE models. We investigate two approaches for this purpose: (1) Single-task Deep Neural Architecture Search (ST-NAS), and (2) Multitask NAS (MT-NAS). In the former approach, personalized models for each learner profile are learned from multimodal features and compared to non-personalized models. In the MT-NAS approach, we investigate whether jointly classifying the learners' profiles with the engagement score through multi-task learning would serve as an implicit personalization of PE. Moreover, we compare the predictive power of two types of features: incremental and non-incremental features. Non-incremental features correspond to features computed from the participant's behaviours in fixed time windows. Incremental features are computed by accounting to the behaviour from the beginning of the learning activity till the time window where productive engagement is observed. Our experimental results show that (1) personalized models improve the recognition performance with respect to non-personalized models when training models for the gainer vs. non-gainer groups, (2) multitask NAS (implicit personalization) also outperforms non-personalized models, (3) the speech modality has high contribution towards prediction, and (4) non-incremental features outperform the incremental ones overall.
KW - Embodied Interaction
KW - Engagement Prediction
KW - Human-robot/Agent Interaction
KW - Personalization
KW - Personalized Affective Computing
KW - Social Robotics in Education
KW - Social Signals
UR - http://www.scopus.com/inward/record.url?scp=85142824193&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85142824193&partnerID=8YFLogxK
U2 - 10.1145/3536221.3556569
DO - 10.1145/3536221.3556569
M3 - Conference contribution
AN - SCOPUS:85142824193
T3 - ACM International Conference Proceeding Series
SP - 632
EP - 641
BT - ICMI 2022 - Proceedings of the 2022 International Conference on Multimodal Interaction
PB - Association for Computing Machinery
T2 - 24th ACM International Conference on Multimodal Interaction, ICMI 2022
Y2 - 7 November 2022 through 11 November 2022
ER -