IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v48y2024i3p495-514.html
   My bibliography  Save this article

How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations

Author

Listed:
  • Burt S. Barnow
  • Sanjay K. Pandey
  • Qian “Eric†Luo

Abstract

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.

Suggested Citation

  • Burt S. Barnow & Sanjay K. Pandey & Qian “Eric†Luo, 2024. "How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations," Evaluation Review, , vol. 48(3), pages 495-514, June.
    • Handle: RePEc:sae:evarev:v:48:y:2024:i:3:p:495-514
    DOI: 10.1177/0193841X241227480
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X241227480
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X241227480?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Chelsea Richwine & Qian Eric Luo & Zoë Thorkildsen & Nicholas J. Chong & Rebecca Morris & Burt S. Barnow & Sanjay K. Pandey, 2022. "Defining And Assessing The Value Of Canonical Mixed Methods Research Designs In Public Policy And Public Administration," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 41(3), pages 891-920, June.
    2. Rao, Vijayendra & Ananthpur, Kripa & Malik, Kabir, 2017. "The Anatomy of Failure: An Ethnography of a Randomized Trial to Deepen Democracy in Rural India," World Development, Elsevier, vol. 99(C), pages 481-497.
    3. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    4. Paul J. Gertler & Sebastian Martinez & Patrick Premand & Laura B. Rawlings & Christel M. J. Vermeersch, 2016. "Impact Evaluation in Practice, Second Edition," World Bank Publications - Books, The World Bank Group, number 25030.
    5. Imbens, Guido W., 2018. "Comments On: Understanding and Misunderstanding Randomized Controlled Trails by Cartwright and Deaton," Research Papers 3648, Stanford University, Graduate School of Business.
    6. Barnow, Burt S., 2010. "Setting up social experiments: the good, the bad, and the ugly," Zeitschrift für ArbeitsmarktForschung - Journal for Labour Market Research, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany], vol. 43(2), pages 91-105.
    7. Martha A. Starr, 2014. "Qualitative And Mixed-Methods Research In Economics: Surprising Growth, Promising Future," Journal of Economic Surveys, Wiley Blackwell, vol. 28(2), pages 238-264, April.
    8. David Greenberg & Karl Ashworth & Andreas Cebulla & Robert Walker, 2005. "When Welfare-to-Work Programs Seem to Work Well: Explaining Why Riverside and Portland Shine So Brightly," ILR Review, Cornell University, ILR School, vol. 59(1), pages 34-50, October.
    9. Edward E. Leamer, 2010. "Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 31-46, Spring.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    2. Sandeep Devanatha Pillai & Brent Goldfarb & David Kirsch, 2024. "Lovely and likely: Using historical methods to improve inference to the best explanation in strategy," Strategic Management Journal, Wiley Blackwell, vol. 45(8), pages 1539-1566, August.
    3. Jean Cartier-Bresson, 2013. "Le pouvoir du positivisme et ses limites : microéconométrie et macroéconométrie actuelles du développement," Working Papers hal-00847005, HAL.
    4. Cyrus J. DiCiccio & Joseph P. Romano & Michael Wolf, 2016. "Improving weighted least squares inference," ECON - Working Papers 232, Department of Economics - University of Zurich, revised Nov 2017.
    5. Heath, Davidson & Ringgenberg, Matthew C. & Samadi, Mehrdad & Werner, Ingrid M., 2019. "Reusing Natural Experiments," Working Paper Series 2019-21, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    6. Romano, Joseph P. & Wolf, Michael, 2017. "Resurrecting weighted least squares," Journal of Econometrics, Elsevier, vol. 197(1), pages 1-19.
    7. Scott Alan Carson, 2013. "The Significance and Relative Contributions of Demographic, Residence, and Socioeconomic Status in Nineteenth-Century U.S. BMI Variation," Historical Methods: A Journal of Quantitative and Interdisciplinary History, Taylor & Francis Journals, vol. 46(2), pages 67-76, June.
    8. Scott A. Carson, 2015. "The Weight of Inequality: Variation with Industrialization and Wealth," CESifo Working Paper Series 5629, CESifo.
    9. Carlianne Patrick & Amanda Ross & Heather Stephens, 2016. "Designing Policies to Spur Economic Growth: How Regional Scientists Can Contribute to Future Policy Development and Evaluation," Working Papers 16-04, Department of Economics, West Virginia University.
    10. Denis Cogneau, 2016. "History, Data and Economics for Africa: Can We Get Them Less Wrong?: Reply to Morten Jerven's ‘Trapped between tragedies and miracles: Misunderstanding African economic growth’," Development Policy Review, Overseas Development Institute, vol. 34(6), pages 895-899, November.
    11. P. Dorian Owen, 2017. "Evaluating Ingenious Instruments for Fundamental Determinants of Long-Run Economic Growth and Development," Econometrics, MDPI, vol. 5(3), pages 1-33, September.
    12. Ashley, Richard A. & Parmeter, Christopher F., 2015. "When is it justifiable to ignore explanatory variable endogeneity in a regression model?," Economics Letters, Elsevier, vol. 137(C), pages 70-74.
    13. Monnet, Eric & Velde, François R., 2020. "Money, Banking, and Old-School Historical Economics," CEPR Discussion Papers 15348, C.E.P.R. Discussion Papers.
    14. van Dijk, M.A., 2014. "The Social Value of Finance," ERIM Inaugural Address Series Research in Management EIA-2014-055-F&A, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam..
    15. Michael Peneder & Martin Woerter, 2014. "Competition, R&D and innovation: testing the inverted-U in a simultaneous system," Journal of Evolutionary Economics, Springer, vol. 24(3), pages 653-687, July.
    16. Gassen, Joachim, 2014. "Causal inference in empirical archival financial accounting research," Accounting, Organizations and Society, Elsevier, vol. 39(7), pages 535-544.
    17. Chaudhuri, Saraswata & Renault, Eric, 2023. "Efficient estimation of regression models with user-specified parametric model for heteroskedasticty," The Warwick Economics Research Paper Series (TWERPS) 1473, University of Warwick, Department of Economics.
    18. Marion Fourcade & Etienne Ollion & Yann Algan, 2015. "La superioridad de los economistas," Revista de Economía Institucional, Universidad Externado de Colombia - Facultad de Economía, vol. 17(33), pages 13-43, July-Dece.
    19. Boris Salazar-Trujillo & Daniel Otero Robles, 2019. "La revolución empírica en economía," Apuntes del Cenes, Universidad Pedagógica y Tecnológica de Colombia, vol. 38(68), pages 15-48, July.
    20. Ralitza Dimova, 2019. "A Debate that Fatigues…: To Randomise or Not to Randomise; What’s the Real Question?," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 31(2), pages 163-168, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:48:y:2024:i:3:p:495-514. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.