Detection of auditory (cross-spectral) and auditory-visual (cross-modal) synchrony

Ken W. Grant, Virginie Van Wassenhove, David Poeppel

Research output: Contribution to journalArticle

Abstract

Detection thresholds for temporal synchrony in auditory and auditory-visual sentence materials were obtained on normal-hearing subjects. For auditory conditions, thresholds were determined using an adaptive-tracking procedure to control the degree of temporal asynchrony of a narrow audio band of speech, both positive and negative in separate tracks, relative to three other narrow audio bands of speech. For auditory-visual conditions, thresholds were determined in a similar manner for each of four narrow audio bands of speech as well as a broadband speech condition, relative to a video image of a female speaker. Four different auditory filter conditions, as well as a broadband auditory-visual speech condition, were evaluated in order to determine whether detection thresholds were dependent on the spectral content of the acoustic speech signal. Consistent with previous studies of auditory-visual speech recognition which showed a broad, asymmetrical range of temporal synchrony for which intelligibility was basically unaffected (audio delays roughly between -40ms and +240 ms), auditory-visual synchrony detection thresholds also showed a broad, asymmetrical pattern of similar magnitude (audio delays roughly between -45ms and +200 ms). No differences in synchrony thresholds were observed for the different filtered bands of speech, or for broadband speech. In contrast, detection thresholds for audio-alone conditions were much smaller (between -17ms and +23ms) and symmetrical. These results suggest a fairly tight coupling between a subject's ability to detect cross-spectral (auditory) and cross-modal (auditory-visual) asynchrony and the intelligibility of auditory and auditory-visual speech materials. Published by Elsevier B.V.

Original languageEnglish (US)
Pages (from-to)43-53
Number of pages11
JournalSpeech Communication
Volume44
Issue number1-4 SPEC. ISS.
DOIs
StatePublished - Oct 2004

Keywords

  • Auditory-visual speech processing
  • Cross-modal asynchrony
  • Spectro-temporal asynchrony

ASJC Scopus subject areas

  • Software
  • Modeling and Simulation
  • Communication
  • Language and Linguistics
  • Linguistics and Language
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Detection of auditory (cross-spectral) and auditory-visual (cross-modal) synchrony'. Together they form a unique fingerprint.

  • Cite this