TY - GEN
T1 - A lightweight combinatorial approach for inferring the ground truth from multiple annotators
AU - Liu, Xiang
AU - Li, Liyun
AU - Memon, Nasir
PY - 2013
Y1 - 2013
N2 - With the increasing importance of producing large-scale labeled datasets for training, testing and validation, services such as Amazon Mechanical Turk (MTurk) are becoming more and more popular to replace the tedious task of manual labeling finished by hand. However, annotators in these crowdsourcing services are known to exhibit different levels of skills, consistencies and even biases, making it difficult to estimate the ground truth class label from the imperfect labels provided by these annotators. To solve this problem, we present a discriminative approach to infer the ground truth class labels by mapping both annotators and the tasks into a low-dimensional space. Our proposed model is inherently combinatorial and therefore does not require any prior knowledge about the annotators or the examples, thereby providing more simplicity and computational efficiency than the state-of-the-art Bayesian methods. We also show that our lightweight approach is, experimentally on real datasets, more accurate than either majority voting or weighted majority voting.
AB - With the increasing importance of producing large-scale labeled datasets for training, testing and validation, services such as Amazon Mechanical Turk (MTurk) are becoming more and more popular to replace the tedious task of manual labeling finished by hand. However, annotators in these crowdsourcing services are known to exhibit different levels of skills, consistencies and even biases, making it difficult to estimate the ground truth class label from the imperfect labels provided by these annotators. To solve this problem, we present a discriminative approach to infer the ground truth class labels by mapping both annotators and the tasks into a low-dimensional space. Our proposed model is inherently combinatorial and therefore does not require any prior knowledge about the annotators or the examples, thereby providing more simplicity and computational efficiency than the state-of-the-art Bayesian methods. We also show that our lightweight approach is, experimentally on real datasets, more accurate than either majority voting or weighted majority voting.
KW - Annotator-Task Model
KW - Combinatorial Model
KW - Crowdsouring
KW - Social Computing
KW - Weighted Majority Voting
UR - http://www.scopus.com/inward/record.url?scp=84881253772&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84881253772&partnerID=8YFLogxK
U2 - 10.1007/978-3-642-39712-7_47
DO - 10.1007/978-3-642-39712-7_47
M3 - Conference contribution
AN - SCOPUS:84881253772
SN - 9783642397110
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 616
EP - 628
BT - Machine Learning and Data Mining in Pattern Recognition - 9th International Conference, MLDM 2013, Proceedings
T2 - 9th International Conference on International Conference on Machine Learning and Data Mining, MLDM 2013
Y2 - 19 July 2013 through 25 July 2013
ER -