Credible evidence on complex change processes: key challenges in impact evaluation on agricultural value chains

Although a growing field of policy intervention, the effectiveness of public-private value chain support is regularly questioned in the policy realm. Partly resulting from stronger pressures on aid money to show its worth, convincing evidence is asked for the effect on poverty alleviation. However, impact evaluations of interventions are challenging: outcome indicators are often multi-dimensional, impact is generated in dynamic and open systems and the external validity of conclusions are often limited, due to contextual particularities. Therefore, there is a strong case for theory-based evaluation where logic models indicate how the intervention is expected to influence the incentives for people’s behaviour. The key assumptions inherent in these casual models can be tested through observation and measurement of specific outcome indicators, using mixed methods in triangulation. The mix of methods will have to anticipate the major threats to validity to the type of evaluative conclusion that the evaluation is expected to generate .Following the work of Shadish, Cook and Campbell (2002), validity threats relate to: 1) statistical conclusion validity; 2) internal validity; 3) construct validity; and, 4) external validity. The authors propose the combined use of data-set observations and causal-process observations in a comparative case-study design, based on critical realist concept of contextmechanism-outcome configurations. The use of a realist method to describe and analyze intervention pilots, facilitates the exchange of experiences between development agencies with evidence-based research. Its defined generalisation domain may prevent uncritical embracement of good practices. Certain value chain upgrading strategies may be viable and effective in a range of situations but are not the panacea, the standard solution, for creating market access; they all involve specific institutional arrangements that ‘fire’ specific mechanisms and incentives that depend on the institutional environment and social capital of stakeholders involved.