TY - JOUR
T1 - Site selection bias in program evaluation
AU - Allcott, Hunt
N1 - Publisher Copyright:
© The Author(s) 2015.
PY - 2015/8/1
Y1 - 2015/8/1
N2 - "Site selection bias" can occur when the probability that a program is adopted or evaluated is correlated with its impacts. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials involving 8.6 million households across the United States. Predictions based on rich microdata from the first 10 replications substantially overstate efficacy in the next 101 sites. Several mechanisms caused this positive selection. For example, utilities in more environmentalist areas are more likely to adopt the program, and their customers are more responsive to the treatment. Also, because utilities initially target treatment at higher-usage consumer subpopulations, efficacy drops as the program is later expanded. The results illustrate how program evaluations can still give systematically biased out-of-sample predictions, even after many replications.
AB - "Site selection bias" can occur when the probability that a program is adopted or evaluated is correlated with its impacts. I test for site selection bias in the context of the Opower energy conservation programs, using 111 randomized control trials involving 8.6 million households across the United States. Predictions based on rich microdata from the first 10 replications substantially overstate efficacy in the next 101 sites. Several mechanisms caused this positive selection. For example, utilities in more environmentalist areas are more likely to adopt the program, and their customers are more responsive to the treatment. Also, because utilities initially target treatment at higher-usage consumer subpopulations, efficacy drops as the program is later expanded. The results illustrate how program evaluations can still give systematically biased out-of-sample predictions, even after many replications.
UR - http://www.scopus.com/inward/record.url?scp=84939491193&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84939491193&partnerID=8YFLogxK
U2 - 10.1093/qje/qjv015
DO - 10.1093/qje/qjv015
M3 - Article
AN - SCOPUS:84939491193
VL - 130
SP - 1117
EP - 1165
JO - Quarterly Journal of Economics
JF - Quarterly Journal of Economics
SN - 0033-5533
IS - 3
ER -