Toward a more nuanced approach to program effectiveness assessment: Hierarchical linear models in K-12 program evaluation

Most K-12 evaluations are designed to make inferences about how a program implemented at the classroom or school level affects student learning outcomes and such inferences inherently involve hierarchical data structure. One methodological challenge for evaluators is linking program implementation factors typically measured at the classroom or teacher level with student outcomes measured at the individual student level. Hierarchical linear modeling (HLM) is ideal for K-12 program evaluations because it can appropriately handle hierarchical data while allowing for a nuanced conceptualization of evaluation questions. The authors illustrate the advantages of HLM with an example of the evaluation of a technology-based reading program for elementary students. HLM enabled evaluators to have a deeper understanding of the relationship between program implementation and program outcomes by (a) providing a better and more proper estimate of the average program outcome, (b) providing an estimate of variation in student outcomes within classrooms, between classrooms, and between schools, and (c) most importantly, by providing a framework for probing what factors are related to variations in student outcomes. Through illustrating the potentials of HLM, the authors aim to advance and expand evaluative tools for establishing empirical evidence regarding program effectiveness in K-12 settings. The authors also hope to bring balance and additional insight into the current methodological discussions about the relationship between program implementation and program impact, rather than merely demonstrating the average effect of a program or policy intervention.