Impact evaluation in field settings: experience from a complex NGO programme in Ethiopia

Evidence from impact evaluations of community-based nutrition interventions addressing the question of 'what works' to reduce linear growth retardation (stunting) is growing but consensus is mixed. Whilst the number of such impact evaluations is increasing, especially by non-governmental organisations, many studies fail to be publicised due to unclear and/or limited positive outcomes, denying experiences of lessons learnt. Those that are published need to be scrutinised to understand the methods used in deciding the validity of the reported outcomes, since impact evaluations in operation settings can suffer a number of limitations. The author uses as an example an integrated community-based non-randomised controlled impact evaluation to highlight some of the problems and issues encountered in carrying out impact evaluations in operational settings and how results need careful interpretation. In this Save the Children UK study, the only intervention to show a positive significant impact in reducing linear growth retardation (stunting) in children 6-36 months was water, sanitation and hygiene. A comprehensive analysis of the nature, reliability and robustness of the study was necessary to determine the validity of the results. This article looks at the important factors that may have been influential in these outcomes, including key issues generally faced in impact evaluations that threaten both the external and internal validity through bias, confounding, contamination and spillover effects as well as implementation problems encountered. The main issues in this evaluation were lack of randomisation and power, alongside implementation issues, resulting in potential confounding and bias possibly producing spurious results. Agencies involved in community-based impact evaluations should engage in robust methods and systematic reporting to create a stronger evidence base and as such be prepared to increase the transparency of their work.