TY - JOUR
T1 - The barking dog that bites
T2 - Test score volatility and school rankings in Punjab, Pakistan
AU - Barrera-Osorio, Felipe
AU - Ganimian, Alejandro J.
N1 - Funding Information:
We gratefully acknowledge the funding provided by the World Bank and the Multidisciplinary Program in Inequality and Social Policy for this project. We thank Dhushyanth Raju, Senior Economist in the South Asia Education Department of the World Bank, who provided the administrative data used in this paper. We also thank Tolani Britton, Silvia Díaz-Granados, Andrew Ho, Dan Koretz, Dick Murnane, Tim Sass, Adela Soliz, and the participants of the Inequality and Social Policy, Quantitative Policy Analysis in Education, and Methods of Educational Measurement seminars at Harvard for their comments. The usual disclaimers apply.
Publisher Copyright:
© 2016 Elsevier Ltd.
PY - 2016/7/1
Y1 - 2016/7/1
N2 - This paper is the first to explore the implications of test score volatility for school accountability policies in a lower-middle-income country. Using two new datasets from the province of Punjab, Pakistan, we find that rankings based on school-level averages of students' test scores fluctuate considerably from one year to the next due to factors unrelated to school quality. First, most variation in test scores is within, not between schools, allowing noise to play a large part in determining school rankings. Second, variation in school test scores is partially attributable to fluctuations in the characteristics of student cohorts. Third, an even larger share is explained by one-time shocks. Fourth, these problems are more pronounced in small schools. As a result, rankings that rely on test score levels or changes often fail to identify top-performing or rapidly-improving schools, since they are influenced by factors beyond schools' control.
AB - This paper is the first to explore the implications of test score volatility for school accountability policies in a lower-middle-income country. Using two new datasets from the province of Punjab, Pakistan, we find that rankings based on school-level averages of students' test scores fluctuate considerably from one year to the next due to factors unrelated to school quality. First, most variation in test scores is within, not between schools, allowing noise to play a large part in determining school rankings. Second, variation in school test scores is partially attributable to fluctuations in the characteristics of student cohorts. Third, an even larger share is explained by one-time shocks. Fourth, these problems are more pronounced in small schools. As a result, rankings that rely on test score levels or changes often fail to identify top-performing or rapidly-improving schools, since they are influenced by factors beyond schools' control.
KW - Accountability
KW - School rankings
KW - Test score volatility
UR - http://www.scopus.com/inward/record.url?scp=84959116070&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84959116070&partnerID=8YFLogxK
U2 - 10.1016/j.ijedudev.2016.01.004
DO - 10.1016/j.ijedudev.2016.01.004
M3 - Article
AN - SCOPUS:84959116070
SN - 0738-0593
VL - 49
SP - 31
EP - 54
JO - International Journal of Educational Development
JF - International Journal of Educational Development
ER -