Scheduling human intelligence tasks in multi-Tenant crowd-powered systems

Djellel Eddine Difallah, Gianluca Demartini, Philippe Cudré-Mauroux

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Micro-Task crowdsourcing has become a popular approach to efiectively tackle complex data management problems such as data linkage, missing values, or schema matching. However, the backend crowdsourced operators of crowd-powered systems typically yield higher latencies than the machineprocessable operators, this is mainly due to inherent efficiency difierences between humans and machines. This problem can be further exacerbated by the lack of workers on the target crowdsourcing platform, or when the workers are shared unequally among a number of competing requesters; including the concurrent users from the same organization who execute crowdsourced queries with difierent types, priorities and prices. Under such conditions, a crowd-powered system acts mostly as a proxy to the crowdsourcing platform, and hence it is very difficult to provide effiency guarantees to its end-users. Scheduling is the traditional way of tackling such problems in computer science, by prioritizing access to shared resources. In this paper, we propose a new crowdsourcing system architecture that leverages scheduling algorithms to optimize task execution in a shared resources environment, in this case a crowdsourcing platform. Our study aims at assessing the efficiency of the crowd in settings where multiple types of tasks are run concurrently. We present extensive experimental results comparing i) difierent multi-Tenant crowdsourcing jobs, including a workload derived from real traces, and ii) difierent scheduling techniques tested with real crowd workers. Our experimental results show that task scheduling can be leveraged to achieve fairness and reduce query latency in multi-Tenant crowd-powered systems, although with very different tradeoffs compared to traditional settings not including human factors.

Original languageEnglish (US)
Title of host publication25th International World Wide Web Conference, WWW 2016
PublisherInternational World Wide Web Conferences Steering Committee
Pages855-865
Number of pages11
ISBN (Electronic)9781450341431
DOIs
StatePublished - 2016
Event25th International World Wide Web Conference, WWW 2016 - Montreal, Canada
Duration: Apr 11 2016Apr 15 2016

Publication series

Name25th International World Wide Web Conference, WWW 2016

Other

Other25th International World Wide Web Conference, WWW 2016
CountryCanada
CityMontreal
Period4/11/164/15/16

Keywords

  • Crowd-Powered System
  • Crowdsourcing
  • Scheduling

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Software

Fingerprint Dive into the research topics of 'Scheduling human intelligence tasks in multi-Tenant crowd-powered systems'. Together they form a unique fingerprint.

Cite this