TY - JOUR
T1 - Epistemic Parity
T2 - Reproducibility as an Evaluation Metric for Differential Privacy
AU - Rosenblatt, Lucas
AU - Herman, Bernease
AU - Holovenko, Anastasia
AU - Lee, Wonkwon
AU - Loftus, Joshua
AU - McKinnie, Elizabeth
AU - Rumezhak, Taras
AU - Stadnik, Andrii
AU - Howe, Bill
AU - Stoyanovich, Julia
N1 - Publisher Copyright:
© 2024 Copyright is held by the owner/author(s).
PY - 2024/5/14
Y1 - 2024/5/14
N2 - Differential privacy (DP) data synthesizers are increasingly proposed to afford public release of sensitive information, offering theoretical guarantees for privacy (and, in some cases, utility), but limited empirical evidence of utility in practical settings. Utility is typically measured as the error on representative proxy tasks, such as descriptive statistics, multivariate correlations, the accuracy of trained classifiers, or performance over a query workload. The ability for these results to generalize to practitioners' experience has been questioned in a number of settings, including the U.S. Census. In this paper, we propose an evaluation methodology for synthetic data that avoids assumptions about the representativeness of proxy tasks, instead measuring the likelihood that published conclusions would change had the authors used synthetic data, a condition we call epistemic parity. Our methodology consists of reproducing empirical conclusions of peer-reviewed papers on real, publicly available data, then re-running these experiments a second time on DP synthetic data and comparing the results.
AB - Differential privacy (DP) data synthesizers are increasingly proposed to afford public release of sensitive information, offering theoretical guarantees for privacy (and, in some cases, utility), but limited empirical evidence of utility in practical settings. Utility is typically measured as the error on representative proxy tasks, such as descriptive statistics, multivariate correlations, the accuracy of trained classifiers, or performance over a query workload. The ability for these results to generalize to practitioners' experience has been questioned in a number of settings, including the U.S. Census. In this paper, we propose an evaluation methodology for synthetic data that avoids assumptions about the representativeness of proxy tasks, instead measuring the likelihood that published conclusions would change had the authors used synthetic data, a condition we call epistemic parity. Our methodology consists of reproducing empirical conclusions of peer-reviewed papers on real, publicly available data, then re-running these experiments a second time on DP synthetic data and comparing the results.
UR - http://www.scopus.com/inward/record.url?scp=85193483963&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85193483963&partnerID=8YFLogxK
U2 - 10.1145/3665252.3665267
DO - 10.1145/3665252.3665267
M3 - Article
AN - SCOPUS:85193483963
SN - 0163-5808
VL - 53
SP - 65
EP - 74
JO - SIGMOD Record
JF - SIGMOD Record
IS - 1
ER -