The effects of school‐based decision‐making on educational outcomes in low‐ and middle‐income contexts: a systematic review

This Campbell systematic review assesses the effectiveness of school‐based decision‐making. The review summarises findings from 17 impact studies and nine studies of barriers and enablers. School‐based decision‐making has small effects in reducing dropouts and repetition. There is a moderate positive effect on average test scores, though the effects are smaller for language and maths. The effects are not large, but comparable to those found in many other effective educational interventions. The positive impact is found in middle‐income countries, with no significant effect in lowincome countries. School‐based decision‐making reforms appear to have a stronger impact on wealthier students with more educated parents, and for children in younger grade levels. School‐based decision‐making reforms appear to be less effective in disadvantaged communities, particularly if parents and community members have low levels of education and low status relative to school personnel. Plain language summary SCHOOL‐BASED DECISION‐MAKING HAS POSITIVE EFFECTS ON EDUCATION OUTCOMES – BUT LESS SO IN LOW‐INCOME COUNTRIES Decentralising decision‐making to schools has small to moderate positive effects in reducing repetition and dropouts, and increasing test scores. These effects are mainly restricted to middle‐income countries, with fewer and smaller positive effects found in low‐income countries or disadvantaged communities. WHAT DID THE REVIEW STUDY? Many governments have addressed the low quality of education by devolving decision‐making authority to schools. It is assumed that locating decision‐making authority within schools will increase accountability, efficiency and responsiveness to local needs. However, there is limited evidence of the effectiveness of these reforms, especially from low‐income countries. Existing reviews on school‐based decision‐making have tended to focus on proximal outcomes and offer very little information about why school‐based decision‐making has positive or negative effects in different circumstances. This review addresses two questions: 1. What is the impact of school‐based decision‐making on educational outcomes in low‐ and middle‐income countries (L&MICs)? 2. What are the barriers to, and enablers of, effective models of school‐based decision‐making? What studies are included? Included studies for the analysis of impact evaluated the change in decision‐making authority from a higher level of decision‐making authority to the level of the school on educational outcomes. Outcomes were either proximal, for example attrition, equality of access, increased enrolment, or final, for example test scores, psychosocial and non‐cognitive skills. Included studies had to have a comparison group and data which were collected since 1990. The analysis of impact included 26 studies, covering 17 interventions. The review identified nine studies to assess barriers and enablers of school‐based decision‐making. What is the aim of this review? This Campbell systematic review assesses the effectiveness of school‐based decision‐making. The review summarises findings from 17 impact studies and nine studies of barriers and enablers. WHAT ARE THE MAIN FINDINGS OF THIS REVIEW? School‐based decision‐making has small effects in reducing dropouts and repetition. There is a moderate positive effect on average test scores, though the effects are smaller for language and maths. The effects are not large, but comparable to those found in many other effective educational interventions. The positive impact is found in middle‐income countries, with no significant effect in low‐income countries. School‐based decision‐making reforms appear to have a stronger impact on wealthier students with more educated parents, and for children in younger grade levels. School‐based decision‐making reforms appear to be less effective in disadvantaged communities, particularly if parents and community members have low levels of education and low status relative to school personnel. WHAT DO THE FINDINGS OF THIS REVIEW MEAN? Implications for policy and practice 1. School‐based decision‐making reforms in highly disadvantaged communities are less likely to be successful. Parental participation seems to be the key to the success of such reforms. 2. The involvement of school management committees in personnel decisions appears to play a role in improving proximal outcomes, such as teacher attendance, but success is also likely to be linked to the overall teacher job market and the prospects of long‐term employment. 3. The specifics of programme design appear to be crucial. Given the limited evidence, we cannot conclude with certainty that incorporating certain elements into school‐based management reforms are generally beneficial. However, it appears that the details of such supplementary elements may be important. Implications for research There needs to be further robust analysis of the impact of large‐scale school‐based decision‐making, as well as further analysis of the conditions that mitigate their impact. There is also a clear need to examine the potentially negative impacts of these reforms, given widespread adoption of such policies. HOW UP‐TO‐DATE IS THIS REVIEW? The review authors searched for studies published until January 2015. This Campbell systematic review was published in November 2016. Executive summary Background Although there have been significant improvements in recent decades, access to education remains limited, particularly for girls, poor children and children in conflict‐affected areas. There is also worrying evidence that many children who are enrolled in school are not learning. Recent estimates suggest that around 130 million children who have completed at least four years of school still cannot read, write or perform basic calculations (UNESCO, 2014, p. 191). Many governments have attempted to address this situation, while also improving efficiency and reducing costs, by devolving decision‐making authority to schools, as it is assumed that locating decision‐making authority within schools will increase accountability, efficiency and responsiveness to local needs (Gertler et al., 2008). This devolution includes a wide variety of models and mechanisms, differing in terms of which decisions are devolved (and how many), to whom decision‐making authority is given, and how the decentralisation process is implemented (i.e., through ‘top‐down’ or ‘bottom‐up' processes). All models and mechanisms are presumed to increase responsiveness to local needs and accountability by bringing community members into direct contact with schools, and to increase efficiency by making financial decisions more transparent to communities, reducing corruption and incentivising investment in high quality teachers and materials. Although the rhetoric around decentralisation suggests that school‐based management has a positive effect on educational outcomes, there is limited evidence from low‐income countries of this general relationship. Existing reviews on school‐based decision‐making have tended to focus on proximal outcomes, while the more comprehensive reviews that do exist are not formal systematic reviews, according to the criteria set by the Campbell Collaboration. They also need updating, as they (a) rely on literature that is now nearly ten years out of date and (b) focus almost exclusively on Central America, referencing almost no evidence from other low‐ and middle‐income countries (L&MICs). Existing reviews on this topic also tell us very little about why school‐based decision‐making has positive or negative effects in different circumstances. Objectives This review aims to address these gaps by answering the following questions: (1) What is the impact of school‐based decision‐making on educational outcomes in low‐ and middle‐income countries (L&MICs) (Review Question 1)? (2) What are the barriers to (and enablers of) effective models of school‐based decision‐making (Review Question 2)? For the purposes of the review, ‘school‐based decision making' was defined as any reform in which decision‐making authority has been devolved to the level of the school. Within this broad definition, there are three main mechanisms discussed in the literature: (1) reforms that devolve decision‐making around management to the school level; (2) reforms that devolve decision‐making around funding to the school level; and (3) reforms that devolve decision‐making around curriculum, pedagogy and other aspects of the classroom environment to the school level. Methods This review followed an explicit protocol following methodological guidance provided by the Campbell Collaboration and the EPPI‐Centre at the UCL Institute of Education (Becker et al., undated; Gough et al., 2012; Hammerstrom, 2009; Shadish & Myers, 2004). To be included in the review, all studies had to: 1) be empirical in nature and focused on primary and secondary schools within L&MICs; 2) investigate a change in decision‐making authority from a higher level of decision‐making authority to the level of the school (excluding studies where the intervention was conceptualised, managed and implemented by an external decision‐making agency, or aimed exclusively at improving the functioning of existing devolved decision‐making structures); 3) provide data on the relationship between school‐based decision‐making and at least one educational outcome (either proximal, e.g. attrition, equality of access, increased enrolment; or final, e.g. student learning, as captured by test scores, psychosocial and non‐cognitive skills, etc.); and 4) rely on data collected since 1990. To be included in reference to Review Question 1, studies needed to be causal in nature, meaning we included: (1) Experimental designs using randomised or quasi‐ randomised assignment; (2) Quasi‐experimental designs; and (3) comparison group designs using before‐and‐after data at baseline and endline, as well as those using cross‐sectional endline data only, where analysis was used to control for confounding. For Review Question 2, we included studies of any empirical design, so long as they provided additional data relating to those interventions featuring in the impact component of the synthesis. Potentially relevant literature was identified through a five‐stage search strategy, which comprised: 1) Identification of existing systematic reviews in related areas; 2) Targeted searches in a wide range of bibliographic databases and websites; 3) Hand searches of the eight most relevant journals relating to the topic; 4) Citation chasing; and 5) Contacting experts involved in the research area. A comprehensive list of search terms was developed in collaboration with information scientists at the EPPI‐Centre. Search terms were also translated into French, Spanish and Portuguese for use in regionally specific databases. All identified literature was subjected to a two‐stage screening process. Relevant studies were then appraised for robustness of evidence and methodological rigour prior to synthesis. In order to answer Review Question 1, we conducted meta‐analysis, relying on the use of ‘standardised mean difference’ (SMD) calculations to compare effects across studies. In our meta‐analysis, we were able to report on the impact of any school‐based decision‐making reform on six educational outcomes: 1) student drop‐out; 2) student repetition; 3) teacher attendance; and 4) student learning, as assessed via i) language test scores, ii) math test scores, iii) aggregate test scores (i.e. tests of more than one subject). We also examined heterogeneity by investigating differences in impacts based on three moderating variables – level of decentralisation, income level, and type of evaluation design. Further, we discuss and synthesise sub‐group effects discussed in the included studies themselves. Analysis in reference to Review Question 2 followed the principles of framework synthesis (Thomas et al., 2012), in order to identify the main barriers and enablers that appear to have influenced the impact of the interventions under review. Results We identified 2,821 titles through our five‐stage search. Of these, 100 met our eligibility criteria. Thirty of the 100 met the design criteria required for RQ1, but three were removed from the RQ1 synthesis, due to high risk of bias. A fourth study had to be excluded due to missing data. Twenty‐six impact studies were thus included in the meta‐analysis. These 26 studies investigate the impact of 17 individual interventions. Of the 73 non‐causal studies subjected to quality appraisal, nine were identified to be of sufficient quality to provide additional data on the included interventions. Devolving decision‐making to the level of the school is found to have a somewhat beneficial effect on drop‐out; a pooled effect of reducing drop‐out by 0.07 standard deviations (SDs). For repetition, the equivalent pooled effect is a reduction of 0.09 SDs. Effects on test‐scores are larger and more robust. We find a positive and significant improvement of 0.21 SDs in aggregate test scores on average, and positive and significant improvements of around 0.07 SDs in scores on language and 0.08 on math tests. Further analysis of test score results suggests that these results pertain to middle income countries, while we did not find statistically significant improvements in test scores in low‐income country settings, with the exception of one study in Kenya (now a middle income country). Evidence does not show that effects on teacher attendance are significant overall, but there is evidence that effects are stronger in contexts of high decentralisation. In common with other comparative studies of the impacts of educational initiatives (Kremer et al., 2013; Snilstveit et al., 2015), these effects of decentralised school‐based decision‐making are relatively small in magnitude. For example, Snilstveit et al. (2015) conducted a recent and broad‐ranging review of interventions to improve learning outcomes in L&MICs and report that the most substantial effects on test‐scores are for ‘structured pedagogy programmes', which found a pooled effect on math scores of 0.14 SDs, while a large number of education intervention types showed no overall effects. Accordingly, while educational effects appear small in comparison to those in some other fields, effects of school‐based decision‐making may be considered similar to interventions that demonstrate medium‐sized effects on education outcomes. Most of the included studies do not conduct any sub‐group analysis relating to individual characteristics, such as gender and student background; those that do differ in their findings. However, there is some evidence to suggest that school‐based decision‐making reforms have a stronger impact on wealthier students with more educated parents. It appears that school‐management reforms may be particularly impactful on children in younger grade levels. School‐based decision‐making reforms appear to be less effective in disadvantaged communities, particularly if parents and community members have low levels of education and low status relative to school personnel. Devolution also appears to be ineffective when communities choose not to actively participate in decision‐making processes. Small schools, however, may find school‐based decision‐making to be effective, particularly if community members establish a collaborative, rather than an adversarial, relationship with teachers. Conclusions and implications for policy, practice and research Overall, we can conclude that devolving decision‐making authority to the school level can have a positive impact on educational outcomes, with magnitudes of effect in the median range for education programmes, but that this is only likely in more advantaged contexts in which community members are largely literate and have sufficient status to participate as equals in the decision‐making process. Our findings carry a number of implications for policy and practice. First, it appears that school‐based decision‐making reforms in highly disadvantaged communities are less likely to be successful. Parental participation seems to be the key to the success of such reforms and this is linked to the real authority or status and cultural capital of community members. Second, the involvement of school management committees in personnel decisions appears to play a role in improving proximal outcomes, such as teacher attendance, but success is also likely to be linked to the overall teacher job market and the prospects of long‐term employment. Third, the specifics of programme design appear to be crucial. Given the limited evidence available in this review, and the contextualised nature of that evidence, we cannot conclude with certainty that incorporating certain elements into school‐based management reforms are generally beneficial. However, it does appear that the details of such supplementary elements may be important. The evidence also suggests that, at least in some contexts, impact on student learning may take longer than is often allowed within evaluation timelines. Where donors are involved, this also means that decentralisation reforms may require sustained donor commitment over the long term. The review also suggests a number of fruitful directions for future research. Although a large number of titles were identified during our initial search, the small number of impact studies included in the meta‐analysis represent a limited geographic diversity and a small number of discrete interventions. There needs to be further robust analysis of the impact(s) of large‐ scale school‐based decision‐making reforms that have recently been implemented, as well as further analysis of the conditions that mitigate their impact. There is also a clear need to examine the potentially negative impacts of these reforms, given widespread adoption of such policies. Although this review has highlighted a number of potential enablers and barriers of effects, the limited evidence base has prevented us from drawing any robust conclusions on the conditions necessary for positive impact. A future review of the same topic, drawing on broader qualitative evidence, would complement the findings of this study.