TY - JOUR
T1 - Deriving gradient measures of child speech from crowdsourced ratings
AU - McAllister Byun, Tara
AU - Harel, Daphna
AU - Halpin, Peter F.
AU - Szeredi, Daniel
N1 - Funding Information:
The authors extend their thanks to Elaine Hitchcock and various lab assistants for their role in collecting and measuring the stimuli used in this study. Aspects of this research were presented at the International Congress of Phonetic Sciences (2015). This project was supported by NIH R03DC 012883.
Publisher Copyright:
© 2016 Elsevier Inc.
PY - 2016/11/1
Y1 - 2016/11/1
N2 - Recent research has demonstrated that perceptual ratings aggregated across multiple non-expert listeners can reveal gradient degrees of contrast between sounds that listeners might transcribe identically. Aggregated ratings have been found to correlate strongly with acoustic gold standard measures both when individual raters use a continuous rating scale such as visual analog scaling (Munson et al., 2012) and when individual raters provide binary ratings (McAllister Byun, Halpin, & Szeredi, 2015). In light of evidence that inexperienced listeners use continuous scales less consistently than experienced listeners, this study investigated the relative merits of binary versus continuous rating scales when aggregating responses over large numbers of naive listeners recruited through online crowdsourcing. Stimuli were words produced by children in treatment for misarticulation of North American English /r/. Each listener rated the same 40 tokens two times: once using Visual Analog Scaling (VAS) and once using a binary rating scale. The gradient rhoticity of each item was then estimated using (a) VAS click location, averaged across raters; (b) the proportion of raters who assigned the “correct /r/” label to each item in the binary rating task (pˆ). First, we validate these two measures of rhoticity against each other and against an acoustic gold standard. Second, we explore the range of variability in individual response patterns that underlie these group-level data. Third, we integrate statistical, theoretical, and practical considerations to offer guidelines for determining which measure to use in a given situation.
AB - Recent research has demonstrated that perceptual ratings aggregated across multiple non-expert listeners can reveal gradient degrees of contrast between sounds that listeners might transcribe identically. Aggregated ratings have been found to correlate strongly with acoustic gold standard measures both when individual raters use a continuous rating scale such as visual analog scaling (Munson et al., 2012) and when individual raters provide binary ratings (McAllister Byun, Halpin, & Szeredi, 2015). In light of evidence that inexperienced listeners use continuous scales less consistently than experienced listeners, this study investigated the relative merits of binary versus continuous rating scales when aggregating responses over large numbers of naive listeners recruited through online crowdsourcing. Stimuli were words produced by children in treatment for misarticulation of North American English /r/. Each listener rated the same 40 tokens two times: once using Visual Analog Scaling (VAS) and once using a binary rating scale. The gradient rhoticity of each item was then estimated using (a) VAS click location, averaged across raters; (b) the proportion of raters who assigned the “correct /r/” label to each item in the binary rating task (pˆ). First, we validate these two measures of rhoticity against each other and against an acoustic gold standard. Second, we explore the range of variability in individual response patterns that underlie these group-level data. Third, we integrate statistical, theoretical, and practical considerations to offer guidelines for determining which measure to use in a given situation.
KW - Covert contrast
KW - Crowdsourcing
KW - Research methods
KW - Speech perception
KW - Speech rating
KW - Speech sound disorders
KW - Visual analog scaling
UR - http://www.scopus.com/inward/record.url?scp=85004125587&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85004125587&partnerID=8YFLogxK
U2 - 10.1016/j.jcomdis.2016.07.001
DO - 10.1016/j.jcomdis.2016.07.001
M3 - Article
C2 - 27481555
AN - SCOPUS:85004125587
SN - 0021-9924
VL - 64
SP - 91
EP - 102
JO - Journal of Communication Disorders
JF - Journal of Communication Disorders
ER -