Data-driven rank aggregation with application to grand challenges

James Fishbaugh, Marcel Prastawa, Bo Wang, Patrick Reynolds, Stephen Aylward, Guido Gerig

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    The increased number of challenges for comparative evaluation of biomedical image analysis procedures clearly reflects a need for unbiased assessment of the state-of-the-art methodological advances. Moreover, the ultimate translation of novel image analysis procedures to the clinic requires rigorous validation and evaluation of alternative schemes, a task that is best outsourced to the international research community. We commonly see an increase of the number of metrics to be used in parallel, reflecting alternative ways to measure similarity. Since different measures come with different scales and distributions, these are often normalized or converted into an individual rank ordering, leaving the problem of combining the set of multiple rankings into a final score. Proposed solutions are averaging or accumulation of rankings, raising the question if different metrics are to be treated the same or if all metrics would be needed to assess closeness to truth. We address this issue with a data-driven method for automatic estimation of weights for a set of metrics based on unsupervised rank aggregation. Our method requires no normalization procedures and makes no assumptions about metric distributions. We explore the sensitivity of metrics to small changes in input data with an iterative perturbation scheme, to prioritize the contribution of the most robust metrics in the overall ranking. We show on real anatomical data that our weighting scheme can dramatically change the ranking.

    Original languageEnglish (US)
    Title of host publicationMedical Image Computing and Computer Assisted Intervention − MICCAI 2017 - 20th International Conference, Proceedings
    EditorsPierre Jannin, Simon Duchesne, Maxime Descoteaux, Alfred Franz, D. Louis Collins, Lena Maier-Hein
    PublisherSpringer Verlag
    Number of pages9
    ISBN (Print)9783319661841
    StatePublished - 2017
    Event20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017 - Quebec City, Canada
    Duration: Sep 11 2017Sep 13 2017

    Publication series

    NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    Volume10434 LNCS
    ISSN (Print)0302-9743
    ISSN (Electronic)1611-3349


    Other20th International Conference on Medical Image Computing and Computer-Assisted Intervention, MICCAI 2017
    CityQuebec City

    ASJC Scopus subject areas

    • Theoretical Computer Science
    • General Computer Science


    Dive into the research topics of 'Data-driven rank aggregation with application to grand challenges'. Together they form a unique fingerprint.

    Cite this