Multi-facial patches aggregation network for facial expression recognition and facial regions contributions to emotion display

Ahmed Rachid Hazourli, Amine Djeghri, Hanan Salam, Alice Othmani

Research output: Contribution to journalArticlepeer-review

Abstract

In this paper, an approach for Facial Expressions Recognition (FER) based on a multi-facial patches (MFP) aggregation network is proposed. Deep features are learned from facial patches using convolutional neural sub-networks and aggregated within one architecture for expression classification. Besides, a framework based on two data augmentation techniques is proposed to expand FER labels training datasets. Consequently, the proposed shallow convolutional neural networks (CNN) based approach does not need large datasets for training. The proposed framework is evaluated on three FER datasets. Results show that the proposed approach achieves state-of-art FER deep learning approaches performance when the model is trained and tested on images from the same dataset. Moreover, the proposed data augmentation techniques improve the expression recognition rate, and thus can be a solution for training deep learning FER models using small datasets. The accuracy degrades significantly when testing for dataset bias. A fine-tuning can overcome the problem of transition from laboratory-controlled conditions to in-the-wild conditions. Finally, the emotional face is mapped using the MFP-CNN and the contribution of the different facial areas in displaying emotion as well as their importance in the recognition of each facial expression are studied.

Original languageEnglish (US)
JournalMultimedia Tools and Applications
DOIs
StateAccepted/In press - 2021

Keywords

  • Conditional generative adversarial network
  • Deep visual learning
  • Facial expression recognition
  • Human-computer interaction
  • multi-facial patches

ASJC Scopus subject areas

  • Software
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'Multi-facial patches aggregation network for facial expression recognition and facial regions contributions to emotion display'. Together they form a unique fingerprint.

Cite this