Bayesian sensitivity analysis for offline policy evaluation

Jongbin Jung, Ravi Shroff, Avi Feller, Sharad Goel

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

On a variety of complex decision-making tasks, from doctors prescribing treatment to judges setting bail, machine learning algorithms have been shown to outperform expert human judgments. One complication, however, is that it is often difficult to anticipate the effects of algorithmic policies prior to deployment, as one generally cannot use historical data to directly observe what would have happened had the actions recommended by the algorithm been taken. A common strategy is to model potential outcomes for alternative decisions assuming that there are no unmeasured confounders (i.e., to assume ignorability). But if this ignorability assumption is violated, the predicted and actual effects of an algorithmic policy can diverge sharply. In this paper we present a flexible Bayesian approach to gauge the sensitivity of predicted policy outcomes to unmeasured confounders. In particular, and in contrast to past work, our modeling framework easily enables confounders to vary with the observed covariates. We demonstrate the efficacy of our method on a large dataset of judicial actions, in which one must decide whether defendants awaiting trial should be required to pay bail or can be released without payment.

Original languageEnglish (US)
Title of host publicationAIES 2020 - Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society
PublisherAssociation for Computing Machinery, Inc
Pages64-70
Number of pages7
ISBN (Electronic)9781450371100
DOIs
StatePublished - Feb 7 2020
Event3rd AAAI/ACM Conference on AI, Ethics, and Society, AIES 2020, co-located with AAAI 2020 - New York, United States
Duration: Feb 7 2020Feb 8 2020

Publication series

NameAIES 2020 - Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society

Conference

Conference3rd AAAI/ACM Conference on AI, Ethics, and Society, AIES 2020, co-located with AAAI 2020
CountryUnited States
CityNew York
Period2/7/202/8/20

Keywords

  • Offline policy evaluation
  • Pretrial risk assessment
  • Sensitivity to unmeasured confounding

ASJC Scopus subject areas

  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Bayesian sensitivity analysis for offline policy evaluation'. Together they form a unique fingerprint.

  • Cite this

    Jung, J., Shroff, R., Feller, A., & Goel, S. (2020). Bayesian sensitivity analysis for offline policy evaluation. In AIES 2020 - Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (pp. 64-70). (AIES 2020 - Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society). Association for Computing Machinery, Inc. https://doi.org/10.1145/3375627.3375822