Crowdsourcing with multi-dimensional trust

Xiangyang Liu, He He, John S. Baras

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We consider a typical crowdsourcing task that aggregates input from multiple workers as a problem in information fusion. To cope with the issue of noisy and sometimes malicious input from workers, trust is used to model workers' expertise. In a multi-domain knowledge learning task, however, using scalar-valued trust to model a worker's performance is not sufficient to reflect the worker's trustworthiness in each of the domains. To address this issue, we propose a probabilistic model to jointly infer multi-dimensional trust of workers, multi-domain properties of questions, and true labels of questions. Our model is very flexible and extensible to incorporate metadata associated with questions. To show that, we further propose two extended models, one of which handles input tasks with real-valued features and the other handles tasks with text features by incorporating topic models. Finally, we evaluate our model on real-world datasets and demonstrate that our model is superior to state-of-the-art and the two extended models have even better performance. In addition, our models can effectively recover trust vectors of workers, which can be very useful in task assignment adaptive to workers' trust in the future. These results can be applied for fusion of information from multiple data sources like sensors, human input, machine learning results, or a hybrid of them.

Original languageEnglish (US)
Title of host publication2015 18th International Conference on Information Fusion, Fusion 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages574-581
Number of pages8
ISBN (Electronic)9780982443866
StatePublished - Sep 14 2015
Event18th International Conference on Information Fusion, Fusion 2015 - Washington, United States
Duration: Jul 6 2015Jul 9 2015

Publication series

Name2015 18th International Conference on Information Fusion, Fusion 2015

Conference

Conference18th International Conference on Information Fusion, Fusion 2015
CountryUnited States
CityWashington
Period7/6/157/9/15

ASJC Scopus subject areas

  • Information Systems
  • Signal Processing
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Crowdsourcing with multi-dimensional trust'. Together they form a unique fingerprint.

  • Cite this

    Liu, X., He, H., & Baras, J. S. (2015). Crowdsourcing with multi-dimensional trust. In 2015 18th International Conference on Information Fusion, Fusion 2015 (pp. 574-581). [7266612] (2015 18th International Conference on Information Fusion, Fusion 2015). Institute of Electrical and Electronics Engineers Inc..