Electrophysiology of Auditory-Visual Speech Integration

Virginie van Wassenhove, Ken W. Grant, David Poeppel

Research output: Contribution to conferencePaperpeer-review


Twenty-six native English Speakers identified auditory (A), visual (V), and congruent and incongruent auditory-visual (AV) syllables while undergoing electroencephalography (EEG) in three experiments. In Experiment 1, unimodal (A, V) and bimodal (AV) stimuli were presented in separate blocks. In Experiment 2, the same stimuli were pseudo-randomized in the same blocks, providing a replication of Experiment 1 while testing the effect of participants' expectancy on the AV condition. In Experiment 3, McGurk fusion (audio/pa/dubbed onto visual/ka/, eliciting the percept/ta/) and combination (audio/ka/dubbed onto visual/pa/) stimuli were tested under visual attention [1]. EEG recordings show early effects of visual influence on auditory evoked-related potentials (P1/N1/P2 complex). Specifically, a robust amplitude reduction of the N1/P2 complex was observed (Experiments 1 and 2) that could not be solely accounted for by attentional effects (Experiment 3). The N1/P2 reduction was accompanied by a temporal facilitation (approximating ~20ms) of the P1/N1and N1/P2 transitions in AV conditions. Additionally, incongruent syllables showed a different profile from congruent AV/ta/over a large latency range (~50 to 350ms post-auditory onset), which was influenced by the accuracy of identification of the visual stimuli presented unimodally. Our results suggest that (i) auditory processing is modulated early on by visual speech inputs, in agreement with an early locus of AV speech interaction, (ii) natural precedence of visual kinematics facilitates auditory speech processing in the time domain, and (iii) the degree of temporal gain is a function of the saliency of visual speech inputs.

Original languageEnglish (US)
Number of pages6
StatePublished - 2003
Event2003 International Conference on Audio-Visual Speech Processing, AVSP 2003 - St. Jorioz, France
Duration: Sep 4 2003Sep 7 2003


Conference2003 International Conference on Audio-Visual Speech Processing, AVSP 2003
CitySt. Jorioz

ASJC Scopus subject areas

  • Language and Linguistics
  • Speech and Hearing
  • Otorhinolaryngology


Dive into the research topics of 'Electrophysiology of Auditory-Visual Speech Integration'. Together they form a unique fingerprint.

Cite this