A lightweight combinatorial approach for inferring the ground truth from multiple annotators

Xiang Liu, Liyun Li, Nasir Memon

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

With the increasing importance of producing large-scale labeled datasets for training, testing and validation, services such as Amazon Mechanical Turk (MTurk) are becoming more and more popular to replace the tedious task of manual labeling finished by hand. However, annotators in these crowdsourcing services are known to exhibit different levels of skills, consistencies and even biases, making it difficult to estimate the ground truth class label from the imperfect labels provided by these annotators. To solve this problem, we present a discriminative approach to infer the ground truth class labels by mapping both annotators and the tasks into a low-dimensional space. Our proposed model is inherently combinatorial and therefore does not require any prior knowledge about the annotators or the examples, thereby providing more simplicity and computational efficiency than the state-of-the-art Bayesian methods. We also show that our lightweight approach is, experimentally on real datasets, more accurate than either majority voting or weighted majority voting.

Original languageEnglish (US)
Title of host publicationMachine Learning and Data Mining in Pattern Recognition - 9th International Conference, MLDM 2013, Proceedings
Pages616-628
Number of pages13
DOIs
StatePublished - 2013
Event9th International Conference on International Conference on Machine Learning and Data Mining, MLDM 2013 - New York, NY, United States
Duration: Jul 19 2013Jul 25 2013

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume7988 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other9th International Conference on International Conference on Machine Learning and Data Mining, MLDM 2013
CountryUnited States
CityNew York, NY
Period7/19/137/25/13

Keywords

  • Annotator-Task Model
  • Combinatorial Model
  • Crowdsouring
  • Social Computing
  • Weighted Majority Voting

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint Dive into the research topics of 'A lightweight combinatorial approach for inferring the ground truth from multiple annotators'. Together they form a unique fingerprint.

  • Cite this

    Liu, X., Li, L., & Memon, N. (2013). A lightweight combinatorial approach for inferring the ground truth from multiple annotators. In Machine Learning and Data Mining in Pattern Recognition - 9th International Conference, MLDM 2013, Proceedings (pp. 616-628). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7988 LNAI). https://doi.org/10.1007/978-3-642-39712-7_47