IDEAS home Printed from https://ideas.repec.org/a/anr/reseco/v15y2023p495-515.html
   My bibliography  Save this article

The Rigor Revolution: New Standards of Evidence for Impact Assessment of International Agricultural Research

Author

Listed:
  • James R. Stevenson

    (CGIAR Standing Panel on Impact Assessment, Alliance of Bioversity International and CIAT, Rome, Italy)

  • Karen Macours

    (Paris School of Economics, Paris, France)

  • Douglas Gollin

    (Department of Economics, University of Oxford, Oxford, United Kingdom)

Abstract

We take stock of the major changes in methodology for studying the impacts of international agricultural research, focusing on the period 2006–2020. Impact assessment of agricultural research has a long and recognized tradition. Until the mid-2000s, such assessments were dominated by a model of demand for and supply of agricultural products in partial equilibrium. The basic ideas for this approach were sketched out by Griliches more than half a century ago. We describe the implications of heightened standards of evidence for good practice in three domains of research design: causal inference, valid measurement, and statistical representativeness. We document advances in each of these domains and review recent evidence that demonstrates the lessons that can be learned from adopting these practices, emphasizing the importance of evidence at-scale, the need to consider portfolios of innovations at a national level, and the challenges of accounting for innovations that are promoted as bundles.

Suggested Citation

  • James R. Stevenson & Karen Macours & Douglas Gollin, 2023. "The Rigor Revolution: New Standards of Evidence for Impact Assessment of International Agricultural Research," Annual Review of Resource Economics, Annual Reviews, vol. 15(1), pages 495-515, October.
  • Handle: RePEc:anr:reseco:v:15:y:2023:p:495-515
    DOI: 10.1146/annurev-resource-101722-082519
    as

    Download full text from publisher

    File URL: https://doi.org/10.1146/annurev-resource-101722-082519
    Download Restriction: Full text downloads are only available to subscribers. Visit the abstract page for more information.

    File URL: https://libkey.io/10.1146/annurev-resource-101722-082519?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Tanguy Bernard & Sylvie Lambert & Karen Macours & Margaux Vinez, 2023. "Impact of small farmers' access to improved seeds and deforestation in DR Congo," Nature Communications, Nature, vol. 14(1), pages 1-14, December.
    2. Howard White, 2009. "Theory-based impact evaluation: principles and practice," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 1(3), pages 271-284.
    3. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    4. Rao, Xudong & Hurley, Terrance M. & Pardey, Philip G., 2020. "Recalibrating the reported returns to agricultural R&D: what if we all heeded Griliches?," Australian Journal of Agricultural and Resource Economics, Australian Agricultural and Resource Economics Society, vol. 64(3), July.
    5. Douglas Gollin & Christopher Udry, 2021. "Heterogeneity, Measurement Error, and Misallocation: Evidence from African Agriculture," Journal of Political Economy, University of Chicago Press, vol. 129(1), pages 1-80.
    6. Dalrymple, Dana G., 1978. "Development and Spread of High-Yielding Varieties of Wheat and Rice in the Less Developed Nations (Sixth Edition)," Foreign Agricultural Economic Report (FAER) 330481, United States Department of Agriculture, Economic Research Service.
    7. Lori Beaman & Ariel BenYishay & Jeremy Magruder & Ahmed Mushfiq Mobarak, 2021. "Can Network Theory-Based Targeting Increase Technology Adoption?," American Economic Review, American Economic Association, vol. 111(6), pages 1918-1943, June.
    8. Rachid Laaja & Karen Macours, 2021. "Measuring Skills in Developing Countries," Journal of Human Resources, University of Wisconsin Press, vol. 56(4), pages 1254-1295.
    9. Rachid Laajaj & Karen Macours & Cargele Masso & Moses Thuita & Bernard Vanlauwe, 2020. "Reconciling yield gains in agronomic trials with returns under African smallholder conditions," PSE-Ecole d'économie de Paris (Postprint) halshs-02973685, HAL.
    10. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    11. Carletto,Calogero & Dillon,Andrew S. & Zezza,Alberto, 2021. "Agricultural Data Collection to Minimize Measurement Error and Maximize Coverage," Policy Research Working Paper Series 9745, The World Bank.
    12. Christopher B. Barrett & Tim G. Benton & Karen A. Cooper & Jessica Fanzo & Rikin Gandhi & Mario Herrero & Steven James & Mark Kahn & Daniel Mason-D’Croz & Alexander Mathys & Rebecca J. Nelson & Jianbo, 2020. "Bundling innovations to transform agri-food systems," Nature Sustainability, Nature, vol. 3(12), pages 974-976, December.
    13. Lant Pritchett, 2002. "It pays to be ignorant: A simple political economy of rigorous program evaluation," Journal of Economic Policy Reform, Taylor & Francis Journals, vol. 5(4), pages 251-269.
    14. repec:hal:pseptp:halshs-03956411 is not listed on IDEAS
    15. Beegle, Kathleen & Carletto, Calogero & Himelein, Kristen, 2012. "Reliability of recall in agricultural data," Journal of Development Economics, Elsevier, vol. 98(1), pages 34-41.
    16. Jeffrey J. Reimer, 2007. "Assessing Global Computable General Equilibrium Model Validity Using Agricultural Price Volatility," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 89(2), pages 383-397.
    17. Hurley, Terrance M. & Pardey, Philip G. & Rao, Xudong & Andrade, Robert S., 2016. "Returns to Food and Agricultural R&D Investments Worldwide, 1958-2015," Briefs 249356, University of Minnesota, International Science and Technology Practice and Policy.
    18. Christopher B. Barrett & Michael R. Carter, 2010. "The Power and Pitfalls of Experiments in Development Economics: Some Non-random Reflections," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 32(4), pages 515-548.
    19. Joachim De Weerdt & John Gibson & Kathleen Beegle, 2020. "What Can We Learn from Experimenting with Survey Methods?," Annual Review of Resource Economics, Annual Reviews, vol. 12(1), pages 431-447, October.
    20. repec:oup:apecpp:v:40:y:2018:i:3:p:421-444. is not listed on IDEAS
    21. Aragón, Fernando M. & Restuccia, Diego & Rud, Juan Pablo, 2024. "Assessing misallocation in agriculture: Plots versus farms," Food Policy, Elsevier, vol. 128(C).
    22. David A Raitzer & Timothy G Kelley, 2008. "Assessing the contribution of impact assessment to donor decisions for international agricultural research," Research Evaluation, Oxford University Press, vol. 17(3), pages 187-199, September.
    23. Kibrom A. Abay & Tesfamicheal Wossen & Gashaw T. Abate & James R. Stevenson & Hope Michelson & Christopher B. Barrett, 2023. "Inferential and Behavioral Implications of Measurement Error in Agricultural Data," Annual Review of Resource Economics, Annual Reviews, vol. 15(1), pages 63-83, October.
    24. Beaman, Lori & Dillon, Andrew, 2018. "Diffusion of agricultural information within social networks: Evidence on gender inequalities from Mali," Journal of Development Economics, Elsevier, vol. 133(C), pages 147-161.
    25. Karen Macours, 2019. "Farmers’ Demand and the Traits and Diffusion of Agricultural Innovations in Developing Countries," Annual Review of Resource Economics, Annual Reviews, vol. 11(1), pages 483-499, October.
    26. Cheryl Doss & Caitlin Kieran & Talip Kilic, 2020. "Measuring Ownership, Control, and Use of Assets," Feminist Economics, Taylor & Francis Journals, vol. 26(3), pages 144-168, July.
    27. Jenny C. Aker & Kelsey Jack, 2021. "Harvesting the Rain: The Adoption of Environmental Technologies in the Sahel," NBER Working Papers 29518, National Bureau of Economic Research, Inc.
    28. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    29. Erwin Corong & Thomas Hertel & Robert McDougall & Marinos Tsigas & Dominique van der Mensbrugghe, 2017. "The Standard GTAP Model, version 7," Journal of Global Economic Analysis, Center for Global Trade Analysis, Department of Agricultural Economics, Purdue University, vol. 2(1), pages 1-119, June.
    30. Alessandro De Pinto & Keith Wiebe & Pablo Pacheco, 2017. "Help bigger palm oil yields to save land," Nature, Nature, vol. 544(7651), pages 416-416, April.
    31. Ruth Stewart & Laurenz Langer & Natalie Rebelo Da Silva & Evans Muchiri & Hazel Zaranyika & Yvonne Erasmus & Nicola Randall & Shannon Rafferty & Marcel Korth & Nolizwe Madinga & Thea de Wet, 2015. "The Effects of Training, Innovation and New Technology on African Smallholder Farmers' Economic Outcomes and Food Security: A Systematic Review," Campbell Systematic Reviews, John Wiley & Sons, vol. 11(1), pages 1-224.
    32. Mark R Rosenzweig & Christopher Udry, 2020. "External Validity in a Stochastic World: Evidence from Low-Income Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 87(1), pages 343-381.
    33. Magnan, Nicholas & Spielman, David J. & Lybbert, Travis J. & Gulati, Kajal, 2015. "Leveling with friends: Social networks and Indian farmers' demand for a technology with heterogeneous benefits," Journal of Development Economics, Elsevier, vol. 116(C), pages 223-251.
    34. Sarah Baird & J. Aislinn Bohren & Craig McIntosh & Berk Özler, 2018. "Optimal Design of Experiments in the Presence of Interference," The Review of Economics and Statistics, MIT Press, vol. 100(5), pages 844-860, December.
    35. Susan W. Parker & Petra E. Todd, 2017. "Conditional Cash Transfers: The Case of Progresa/Oportunidades," Journal of Economic Literature, American Economic Association, vol. 55(3), pages 866-915, September.
    36. Lilyan E. Fulginiti, 2010. "Estimating Griliches' k-Shifts," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 92(1), pages 86-101.
    37. Kyle Emerick & Manzoor H. Dar, 2021. "Farmer Field Days and Demonstrator Selection for Increasing Technology Adoption," The Review of Economics and Statistics, MIT Press, vol. 103(4), pages 680-693, October.
    38. Maredia, Mywish K. & Reyes, Byron A. & Manu-Aduening, Joseph & Dankyi, Awere & Hamazakaza, Petan & Muimui, Kennedy & Rabbi, Ismail & Kulakow, Peter & Parkes, Elizabeth & Abdoulaye, Tahirou & Katungi, , 2016. "Testing Alternative Methods of Varietal Identification Using DNA Fingerprinting: Results of Pilot Studies in Ghana and Zambia," Food Security International Development Working Papers 246950, Michigan State University, Department of Agricultural, Food, and Resource Economics.
    39. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    40. repec:hal:journl:halshs-03956411 is not listed on IDEAS
    41. Erwin Bulte & Gonne Beekman & Salvatore Di Falco & Joseph Hella & Pan Lei, 2014. "Behavioral Responses and the Impact of New Agricultural Technologies: Evidence from a Double-blind Field Experiment in Tanzania," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 96(3), pages 813-830.
    42. Quisumbing, Agnes R. & Ahmed, Akhter & Gilligan, Daniel O. & Hoddinott, John & Kumar, Neha & Leroy, Jef L. & Menon, Purnima & Olney, Deanna K. & Roy, Shalini & Ruel, Marie, 2020. "Randomized controlled trials of multi-sectoral programs: Lessons from development research," World Development, Elsevier, vol. 127(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alemu, Solomon & Kosmowski, Frederic & Stevenson, James R. & Mallia, Paola & Taye, Lemi & Macours, Karen, 2024. "Studying inclusive innovation with the right data: An empirical illustration from Ethiopia," Agricultural Systems, Elsevier, vol. 219(C).
    2. Fuglie, Keith O. & Echeverria, Ruben G., 2024. "The economic impact of CGIAR-related crop technologies on agricultural productivity in developing countries, 1961–2020," World Development, Elsevier, vol. 176(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kazushi Takahashi & Rie Muraoka & Keijiro Otsuka, 2020. "Technology adoption, impact, and extension in developing countries’ agriculture: A review of the recent literature," Agricultural Economics, International Association of Agricultural Economists, vol. 51(1), pages 31-45, January.
    2. Wossen, Tesfamicheal & Abay, Kibrom A. & Abdoulaye, Tahirou, 2022. "Misperceiving and misreporting input quality: Implications for input use and productivity," Journal of Development Economics, Elsevier, vol. 157(C).
    3. Masselus, Lise & Fiala, Nathan, 2024. "Whom to ask? Testing respondent effects in household surveys," Journal of Development Economics, Elsevier, vol. 168(C).
    4. Fiala, Nathan & Masselus, Lise, 2022. "Whom to ask? Testing respondent effects in household surveys," Ruhr Economic Papers 935, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    5. Kabeer, Naila, 2020. "‘Misbehaving’ RCTs: The confounding problem of human agency," World Development, Elsevier, vol. 127(C).
    6. Arslan, Cansın & Wollni, Meike & Oduol, Judith & Hughes, Karl, 2022. "Who communicates the information matters for technology adoption," World Development, Elsevier, vol. 158(C).
    7. Ankel-Peters, Jörg & Schmidt, Christoph M., 2023. "Rural electrification, the credibility revolution, and the limits of evidence-based policy," Ruhr Economic Papers 1051, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    8. Maximilian Kasy & Jann Spiess, 2022. "Optimal Pre-Analysis Plans: Statistical Decisions Subject to Implementability," Papers 2208.09638, arXiv.org, revised Jul 2024.
    9. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," Working Papers hal-03455978, HAL.
    10. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.
    11. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    12. repec:lic:licosd:40718 is not listed on IDEAS
    13. Maximilian Kasy & Jann Spiess, 2024. "Optimal Pre-Analysis Plans: Statistical Decisions Subject to Implementability," Economics Series Working Papers 1058, University of Oxford, Department of Economics.
    14. Michelson, Hope & Gourlay, Sydney & Lybbert, Travis & Wollburg, Philip, 2023. "Review: Purchased agricultural input quality and small farms," Food Policy, Elsevier, vol. 116(C).
    15. Jérémie Gignoux & Karen Macours & Daniel Stein & Kelsey Wright, 2023. "Input subsidies, credit constraints, and expectations of future transfers: Evidence from Haiti," American Journal of Agricultural Economics, John Wiley & Sons, vol. 105(3), pages 809-835, May.
    16. Carletto,Calogero & Dillon,Andrew S. & Zezza,Alberto, 2021. "Agricultural Data Collection to Minimize Measurement Error and Maximize Coverage," Policy Research Working Paper Series 9745, The World Bank.
    17. Anna Dreber & Magnus Johannesson & Yifan Yang, 2024. "Selective reporting of placebo tests in top economics journals," Economic Inquiry, Western Economic Association International, vol. 62(3), pages 921-932, July.
    18. Abel Brodeur, Nikolai M. Cook, Jonathan S. Hartley, Anthony Heyes, 2022. "Do Pre-Registration and Pre-analysis Plans Reduce p-Hacking and Publication Bias?," LCERPA Working Papers am0132, Laurier Centre for Economic Research and Policy Analysis.
    19. Mattea Stein, 2021. "Know-how and Know-who: Effects of a Randomized Training on Network Changes Among Small Urban Entrepreneurs," CSEF Working Papers 622, Centre for Studies in Economics and Finance (CSEF), University of Naples, Italy.
    20. Paola Mallia, 2022. "You reap what (you think) you sow? Evidence on farmers’behavioral adjustments in the case of correct crop varietal identification," PSE Working Papers hal-03597332, HAL.
    21. Michael Carter & Rachid Laajaj & Dean Yang, 2021. "Subsidies and the African Green Revolution: Direct Effects and Social Network Spillovers of Randomized Input Subsidies in Mozambique," American Economic Journal: Applied Economics, American Economic Association, vol. 13(2), pages 206-229, April.

    More about this item

    Keywords

    agricultural research for development; impact assessment; measurement; causality; scale;
    All these keywords.

    JEL classification:

    • O12 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Microeconomic Analyses of Economic Development
    • O13 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Agriculture; Natural Resources; Environment; Other Primary Products
    • O32 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Management of Technological Innovation and R&D
    • O33 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Technological Change: Choices and Consequences; Diffusion Processes
    • Q16 - Agricultural and Natural Resource Economics; Environmental and Ecological Economics - - Agriculture - - - R&D; Agricultural Technology; Biofuels; Agricultural Extension Services

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:anr:reseco:v:15:y:2023:p:495-515. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: http://www.annualreviews.org (email available below). General contact details of provider: http://www.annualreviews.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.