Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system

Marina Zimmermann, Mostafa Mehdipour Ghazi, Hazım Kemal Ekenel, Jean Philippe Thiran

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Automatic visual speech recognition is an interesting problem in pattern recognition especially when audio data is noisy or not readily available. It is also a very challenging task mainly because of the lower amount of information in the visual articulations compared to the audible utterance. In this work, principle component analysis is applied to the image patches — extracted from the video data — to learn the weights of a two-stage convolutional network. Block histograms are then extracted as the unsupervised learning features. These features are employed to learn a recurrent neural network with a set of long short-term memory cells to obtain spatiotemporal features. Finally, the obtained features are used in a tandem GMM-HMM system for speech recognition. Our results show that the proposed method has outperformed the baseline techniques applied to the OuluVS2 audiovisual database for phrase recognition with the frontal view cross-validation and testing sentence correctness reaching 79% and 73%, respectively, as compared to the baseline of 74% on cross-validation.

Original languageEnglish (US)
Title of host publicationComputer Vision - ACCV 2016 Workshops, ACCV 2016 International Workshops, Revised Selected Papers
EditorsKai-Kuang Ma, Jiwen Lu, Chu-Song Chen
PublisherSpringer Verlag
Pages264-276
Number of pages13
ISBN (Print)9783319544267
DOIs
StatePublished - 2017
Event13th Asian Conference on Computer Vision, ACCV 2016 - Taipei, Taiwan, Province of China
Duration: Nov 20 2016Nov 24 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10117 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference13th Asian Conference on Computer Vision, ACCV 2016
Country/TerritoryTaiwan, Province of China
City Taipei
Period11/20/1611/24/16

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Visual speech recognition using PCA networks and LSTMs in a tandem GMM-HMM system'. Together they form a unique fingerprint.

Cite this