Analyzing crowd rankings

Julia Stoyanovich, Marie Jacob, Xuemei Gong

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    Ranked data is ubiquitous in real-world applications, arising naturally when users express preferences about products and services, when voters cast ballots in elections, and when funding proposals are evaluated based on their merits or university departments based on their reputation. This paper focuses on crowd-sourcing and novel analysis of ranked data. We describe the design of a data collection task in which Amazon MT workers were asked to rank movies. We present results of data analysis, correlating our ranked dataset with IMDb, where movies are rated on a discrete scale rather than ranked. We develop an intuitive measure of worker quality appropriate for this task, where no gold standard answer exists. We propose a model of local structure in ranked datasets, reflecting that subsets of the workers agree in their ranking over subsets of the items, develop a data mining algorithm that identifies such structure, and evalu- Ate in on our dataset. Our dataset is publicly available at https://github.com/stoyanovich/CrowdRank.

    Original languageEnglish (US)
    Title of host publication18th International Workshop on the Web and Databases, WebDB 2015
    Subtitle of host publicationFreshness, Correctness, Quality of Information and Knowledge on the Web - Proceedings
    EditorsJulia Stoyanovich, Fabian M. Suchanek
    PublisherAssociation for Computing Machinery, Inc
    Pages41-47
    Number of pages7
    ISBN (Electronic)9781450336277
    DOIs
    StatePublished - May 31 2015
    Event18th International Workshop on the Web and Databases, WebDB 2015 - Melbourne, Australia
    Duration: May 31 2015 → …

    Publication series

    Name18th International Workshop on the Web and Databases, WebDB 2015: Freshness, Correctness, Quality of Information and Knowledge on the Web - Proceedings

    Other

    Other18th International Workshop on the Web and Databases, WebDB 2015
    CountryAustralia
    CityMelbourne
    Period5/31/15 → …

    ASJC Scopus subject areas

    • Computer Networks and Communications
    • Computer Science Applications
    • Information Systems

    Fingerprint Dive into the research topics of 'Analyzing crowd rankings'. Together they form a unique fingerprint.

  • Cite this

    Stoyanovich, J., Jacob, M., & Gong, X. (2015). Analyzing crowd rankings. In J. Stoyanovich, & F. M. Suchanek (Eds.), 18th International Workshop on the Web and Databases, WebDB 2015: Freshness, Correctness, Quality of Information and Knowledge on the Web - Proceedings (pp. 41-47). (18th International Workshop on the Web and Databases, WebDB 2015: Freshness, Correctness, Quality of Information and Knowledge on the Web - Proceedings). Association for Computing Machinery, Inc. https://doi.org/10.1145/2767109.2767110