Epistemic Parity: Reproducibility as an Evaluation Metric for Differential Privacy

Lucas Rosenblatt, Joshua Loftus, Bernease Herman, Elizabeth McKinnie, Bill Howe, Anastasia Holovenko, Taras Rumezhak, Julia Stoyanovich, Wonkwon Lee, Andrii Stadnik

    Research output: Contribution to journalConference articlepeer-review


    Differential privacy (DP) data synthesizers are increasingly proposed to afford public release of sensitive information, offering theoretical guarantees for privacy (and, in some cases, utility), but limited empirical evidence of utility in practical settings. Utility is typically measured as the error on representative proxy tasks, such as descriptive statistics, multivariate correlations, the accuracy of trained classifiers, or performance over a query workload. The ability for these results to generalize to practitioners’ experience has been questioned in a number of settings, including the U.S. Census. In this paper, we propose an evaluation methodology for synthetic data that avoids assumptions about the representativeness of proxy tasks, instead measuring the likelihood that published conclusions would change had the authors used synthetic data, a condition we call epistemic parity. Our methodology consists of reproducing empirical conclusions of peer-reviewed papers on real, publicly available data, then re-running these experiments a second time on DP synthetic data and comparing the results. We instantiate our methodology over a benchmark of recent peer-reviewed papers that analyze public datasets in the ICPSR social science repository. We model quantitative claims computationally to automate the experimental workflow, and model qualitative claims by reproducing visualizations and comparing the results manually. We then generate DP synthetic datasets using multiple state-of-the-art mechanisms, and estimate the likelihood that these conclusions will hold. We find that, for reasonable privacy regimes, state-of-the-art DP synthesizers are able to achieve high epistemic parity for several papers in our benchmark. However, some papers, and particularly some specific findings, are difficult to reproduce for any of the synthesizers. Given these results, we advocate for a new class of mechanisms that can reorder the priorities for DP data synthesis: favor stronger guarantees for utility (as measured by epistemic parity) and offer privacy protection with a focus on application-specific threat models and risk-assessment.

    Original languageEnglish (US)
    Pages (from-to)3178-3191
    Number of pages14
    JournalProceedings of the VLDB Endowment
    Issue number11
    StatePublished - 2023
    Event49th International Conference on Very Large Data Bases, VLDB 2023 - Vancouver, Canada
    Duration: Aug 28 2023Sep 1 2023

    ASJC Scopus subject areas

    • Computer Science (miscellaneous)
    • General Computer Science


    Dive into the research topics of 'Epistemic Parity: Reproducibility as an Evaluation Metric for Differential Privacy'. Together they form a unique fingerprint.

    Cite this