TY - CONF
T1 - Electrophysiology of Auditory-Visual Speech Integration
AU - van Wassenhove, Virginie
AU - Grant, Ken W.
AU - Poeppel, David
N1 - Funding Information:
This work was funded by NIH grant DC 05660 to DP. We would like to thank Jonathan Z. Simon for his help on bootstrapping. . The opinions or assertions contained herein are the private views of the authors [KG] and are not to be construed as official or as reflecting the views of the Department of the Army or the Department of Defense.
Publisher Copyright:
© 2003 AVSP 2003 - International Conference on Audio-Visual Speech Processing. All rights reserved.
PY - 2003
Y1 - 2003
N2 - Twenty-six native English Speakers identified auditory (A), visual (V), and congruent and incongruent auditory-visual (AV) syllables while undergoing electroencephalography (EEG) in three experiments. In Experiment 1, unimodal (A, V) and bimodal (AV) stimuli were presented in separate blocks. In Experiment 2, the same stimuli were pseudo-randomized in the same blocks, providing a replication of Experiment 1 while testing the effect of participants' expectancy on the AV condition. In Experiment 3, McGurk fusion (audio/pa/dubbed onto visual/ka/, eliciting the percept/ta/) and combination (audio/ka/dubbed onto visual/pa/) stimuli were tested under visual attention [1]. EEG recordings show early effects of visual influence on auditory evoked-related potentials (P1/N1/P2 complex). Specifically, a robust amplitude reduction of the N1/P2 complex was observed (Experiments 1 and 2) that could not be solely accounted for by attentional effects (Experiment 3). The N1/P2 reduction was accompanied by a temporal facilitation (approximating ~20ms) of the P1/N1and N1/P2 transitions in AV conditions. Additionally, incongruent syllables showed a different profile from congruent AV/ta/over a large latency range (~50 to 350ms post-auditory onset), which was influenced by the accuracy of identification of the visual stimuli presented unimodally. Our results suggest that (i) auditory processing is modulated early on by visual speech inputs, in agreement with an early locus of AV speech interaction, (ii) natural precedence of visual kinematics facilitates auditory speech processing in the time domain, and (iii) the degree of temporal gain is a function of the saliency of visual speech inputs.
AB - Twenty-six native English Speakers identified auditory (A), visual (V), and congruent and incongruent auditory-visual (AV) syllables while undergoing electroencephalography (EEG) in three experiments. In Experiment 1, unimodal (A, V) and bimodal (AV) stimuli were presented in separate blocks. In Experiment 2, the same stimuli were pseudo-randomized in the same blocks, providing a replication of Experiment 1 while testing the effect of participants' expectancy on the AV condition. In Experiment 3, McGurk fusion (audio/pa/dubbed onto visual/ka/, eliciting the percept/ta/) and combination (audio/ka/dubbed onto visual/pa/) stimuli were tested under visual attention [1]. EEG recordings show early effects of visual influence on auditory evoked-related potentials (P1/N1/P2 complex). Specifically, a robust amplitude reduction of the N1/P2 complex was observed (Experiments 1 and 2) that could not be solely accounted for by attentional effects (Experiment 3). The N1/P2 reduction was accompanied by a temporal facilitation (approximating ~20ms) of the P1/N1and N1/P2 transitions in AV conditions. Additionally, incongruent syllables showed a different profile from congruent AV/ta/over a large latency range (~50 to 350ms post-auditory onset), which was influenced by the accuracy of identification of the visual stimuli presented unimodally. Our results suggest that (i) auditory processing is modulated early on by visual speech inputs, in agreement with an early locus of AV speech interaction, (ii) natural precedence of visual kinematics facilitates auditory speech processing in the time domain, and (iii) the degree of temporal gain is a function of the saliency of visual speech inputs.
UR - http://www.scopus.com/inward/record.url?scp=84919874632&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84919874632&partnerID=8YFLogxK
M3 - Paper
AN - SCOPUS:84919874632
SP - 37
EP - 42
T2 - 2003 International Conference on Audio-Visual Speech Processing, AVSP 2003
Y2 - 4 September 2003 through 7 September 2003
ER -