Fully Automatic Analysis of Engagement and Its Relationship to Personality in Human-Robot Interactions

Hanan Salam, Oya Celiktutan, Isabelle Hupont, Hatice Gunes, Mohamed Chetouani

Research output: Contribution to journalArticlepeer-review


Engagement is crucial to designing intelligent systems that can adapt to the characteristics of their users. This paper focuses on the automatic analysis and classification of engagement based on humans' and robot's personality profiles in a triadic human-human-robot interaction setting. More explicitly, we present a study that involves two participants interacting with a humanoid robot, and investigate how participants' personalities can be used together with the robot's personality to predict the engagement state of each participant. The fully automatic system is first trained to predict the Big Five personality traits of each participant by extracting individual and interpersonal features from their nonverbal behavioural cues. Second, the output of the personality prediction system is used as an input to the engagement classification system. Third, we focus on the concept of 'group engagement', which we define as the collective engagement of the participants with the robot, and analyze the impact of similar and dissimilar personalities on the engagement classification. Our experimental results show that: 1) using the automatically predicted personality labels for engagement classification yields an F-measure on par with using the manually annotated personality labels, demonstrating the effectiveness of the automatic personality prediction module proposed; 2) using the individual and interpersonal features without utilizing personality information is not sufficient for engagement classification, instead incorporating the participants and robots personalities with individual/interpersonal features increases engagement classification performance; and 3) the best classification performance is achieved when the participants and the robot are extroverted, while the worst results are obtained when all are introverted.

Original languageEnglish (US)
Article number7580650
Pages (from-to)705-721
Number of pages17
JournalIEEE Access
StatePublished - 2017


  • Human-robot interaction
  • affective computing
  • engagement classification
  • person-adaptive systems
  • personality prediction

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering


Dive into the research topics of 'Fully Automatic Analysis of Engagement and Its Relationship to Personality in Human-Robot Interactions'. Together they form a unique fingerprint.

Cite this