Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations

Mark Cartwright, Justin Salamon, Ayanna Seals, Oded Nov, Juan Bello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Audio annotation is an important step in developing machine-listening systems. It is also a time consuming process, which has motivated investigators to crowdsource audio annotations. However, there are many factors that affect annotations, many of which have not been adequately investigated. In previous work, we investigated the effects of visualization aids and sound scene complexity on the quality of crowdsourced sound-event annotations. In this paper, we extend that work by investigating the effect of sound-event loudness on both sound-event source annotations and sound-event proximity annotations. We find that the sound class, loudness, and annotator bias affect how listeners annotate proximity. We also find that loudness affects recall more than precision and that the strengths of these effects are strongly influenced by the sound class. These findings are not only important for designing effective audio annotation processes, but also for effectively training and evaluating machine-listening systems.

Original languageEnglish (US)
Title of host publication2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages341-345
Number of pages5
Volume2018-April
ISBN (Print)9781538646588
DOIs
StatePublished - Sep 10 2018
Event2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Calgary, Canada
Duration: Apr 15 2018Apr 20 2018

Other

Other2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018
CountryCanada
CityCalgary
Period4/15/184/20/18

Fingerprint

Acoustic waves
Visualization

Keywords

  • Audio annotations
  • Crowdsourcing
  • Machine listening
  • Sound event detection

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering

Cite this

Cartwright, M., Salamon, J., Seals, A., Nov, O., & Bello, J. (2018). Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations. In 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings (Vol. 2018-April, pp. 341-345). [8461833] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2018.8461833

Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations. / Cartwright, Mark; Salamon, Justin; Seals, Ayanna; Nov, Oded; Bello, Juan.

2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings. Vol. 2018-April Institute of Electrical and Electronics Engineers Inc., 2018. p. 341-345 8461833.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Cartwright, M, Salamon, J, Seals, A, Nov, O & Bello, J 2018, Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations. in 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings. vol. 2018-April, 8461833, Institute of Electrical and Electronics Engineers Inc., pp. 341-345, 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018, Calgary, Canada, 4/15/18. https://doi.org/10.1109/ICASSP.2018.8461833
Cartwright M, Salamon J, Seals A, Nov O, Bello J. Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations. In 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings. Vol. 2018-April. Institute of Electrical and Electronics Engineers Inc. 2018. p. 341-345. 8461833 https://doi.org/10.1109/ICASSP.2018.8461833
Cartwright, Mark ; Salamon, Justin ; Seals, Ayanna ; Nov, Oded ; Bello, Juan. / Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations. 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings. Vol. 2018-April Institute of Electrical and Electronics Engineers Inc., 2018. pp. 341-345
@inproceedings{eb38094e89b14412a3eb0bc2bf9aee3f,
title = "Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations",
abstract = "Audio annotation is an important step in developing machine-listening systems. It is also a time consuming process, which has motivated investigators to crowdsource audio annotations. However, there are many factors that affect annotations, many of which have not been adequately investigated. In previous work, we investigated the effects of visualization aids and sound scene complexity on the quality of crowdsourced sound-event annotations. In this paper, we extend that work by investigating the effect of sound-event loudness on both sound-event source annotations and sound-event proximity annotations. We find that the sound class, loudness, and annotator bias affect how listeners annotate proximity. We also find that loudness affects recall more than precision and that the strengths of these effects are strongly influenced by the sound class. These findings are not only important for designing effective audio annotation processes, but also for effectively training and evaluating machine-listening systems.",
keywords = "Audio annotations, Crowdsourcing, Machine listening, Sound event detection",
author = "Mark Cartwright and Justin Salamon and Ayanna Seals and Oded Nov and Juan Bello",
year = "2018",
month = "9",
day = "10",
doi = "10.1109/ICASSP.2018.8461833",
language = "English (US)",
isbn = "9781538646588",
volume = "2018-April",
pages = "341--345",
booktitle = "2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Investigating the Effect of Sound-Event Loudness on Crowdsourced Audio Annotations

AU - Cartwright, Mark

AU - Salamon, Justin

AU - Seals, Ayanna

AU - Nov, Oded

AU - Bello, Juan

PY - 2018/9/10

Y1 - 2018/9/10

N2 - Audio annotation is an important step in developing machine-listening systems. It is also a time consuming process, which has motivated investigators to crowdsource audio annotations. However, there are many factors that affect annotations, many of which have not been adequately investigated. In previous work, we investigated the effects of visualization aids and sound scene complexity on the quality of crowdsourced sound-event annotations. In this paper, we extend that work by investigating the effect of sound-event loudness on both sound-event source annotations and sound-event proximity annotations. We find that the sound class, loudness, and annotator bias affect how listeners annotate proximity. We also find that loudness affects recall more than precision and that the strengths of these effects are strongly influenced by the sound class. These findings are not only important for designing effective audio annotation processes, but also for effectively training and evaluating machine-listening systems.

AB - Audio annotation is an important step in developing machine-listening systems. It is also a time consuming process, which has motivated investigators to crowdsource audio annotations. However, there are many factors that affect annotations, many of which have not been adequately investigated. In previous work, we investigated the effects of visualization aids and sound scene complexity on the quality of crowdsourced sound-event annotations. In this paper, we extend that work by investigating the effect of sound-event loudness on both sound-event source annotations and sound-event proximity annotations. We find that the sound class, loudness, and annotator bias affect how listeners annotate proximity. We also find that loudness affects recall more than precision and that the strengths of these effects are strongly influenced by the sound class. These findings are not only important for designing effective audio annotation processes, but also for effectively training and evaluating machine-listening systems.

KW - Audio annotations

KW - Crowdsourcing

KW - Machine listening

KW - Sound event detection

UR - http://www.scopus.com/inward/record.url?scp=85054244462&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85054244462&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2018.8461833

DO - 10.1109/ICASSP.2018.8461833

M3 - Conference contribution

SN - 9781538646588

VL - 2018-April

SP - 341

EP - 345

BT - 2018 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2018 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -