Measures of Disparity and their Efficient Estimation

Quantifying disparities, that is differences in outcomes among population groups, is an important task in public health, economics, and increasingly in machine learning. In this work, we study the question of how to collect data to measure disparities. The field of survey statistics provides extensive guidance on sample sizes necessary to accurately estimate quantities such as averages. However, there is limited guidance for estimating disparities. We consider a broad class of disparity metrics including those used in machine learning for measuring fairness of model outputs. For each metric, we derive the number of samples to be collected per group that increases the precision of disparity estimates given a fixed data collection budget. We also provide sample size calculations for hypothesis tests that check for significant disparities. Our methods can be used to determine sample sizes for fairness evaluations. We validate the methods on two nationwide surveys, used for understanding population-level attributes like employment and health, and a prediction model. Absent a priori information on the groups, we find that equally sampling the groups typically performs well.