Crowdsourcing Multi-label Audio Annotation Tasks with Citizen Scientists

Mark Cartwright, Graham Dove, Ana Elisa Méndez Méndez, Juan P. Bello, Oded Nov

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Annotating rich audio data is an essential aspect of training and evaluating machine listening systems. We approach this task in the context of temporally-complex urban soundscapes, which require multiple labels to identify overlapping sound sources. Typically this work is crowdsourced, and previous studies have shown that workers can quickly label audio with binary annotation for single classes. However, this approach can be difcult to scale when multiple passes with diferent focus classes are required to annotate data with multiple labels. In citizen science, where tasks are often image-based, annotation eforts typically label multiple classes simultaneously in a single pass. This paper describes our data collection on the Zooniverse citizen science platform, comparing the efciencies of diferent audio annotation strategies. We compared multiple-pass binary annotation, single-pass multi-label annotation, and a hybrid approach: hierarchical multi-pass multi-label annotation. We discuss our fndings, which support using multi-label annotation, with reference to volunteer citizen scientists’ motivations.

Original languageEnglish (US)
Title of host publicationCHI 2019 - Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450359702
StatePublished - May 2 2019
Event2019 CHI Conference on Human Factors in Computing Systems, CHI 2019 - Glasgow, United Kingdom
Duration: May 4 2019May 9 2019

Publication series

NameConference on Human Factors in Computing Systems - Proceedings


Conference2019 CHI Conference on Human Factors in Computing Systems, CHI 2019
Country/TerritoryUnited Kingdom


  • Audio annotation
  • Citizen science
  • Crowdsourcing

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Crowdsourcing Multi-label Audio Annotation Tasks with Citizen Scientists'. Together they form a unique fingerprint.

Cite this