IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v48y2024i3p427-460.html
   My bibliography  Save this article

The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations

Author

Listed:
  • Julia H. Littell

Abstract

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of “evidence-based†psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.

Suggested Citation

  • Julia H. Littell, 2024. "The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations," Evaluation Review, , vol. 48(3), pages 427-460, June.
    • Handle: RePEc:sae:evarev:v:48:y:2024:i:3:p:427-460
    DOI: 10.1177/0193841X241227481
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X241227481
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X241227481?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Charles F. Manski, 2013. "Response to the Review of ‘Public Policy in an Uncertain World’," Economic Journal, Royal Economic Society, vol. 0, pages 412-415, August.
    2. Kerry Dwan & Douglas G Altman & Mike Clarke & Carrol Gamble & Julian P T Higgins & Jonathan A C Sterne & Paula R Williamson & Jamie J Kirkham, 2014. "Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical Trials: A Systematic Review of Cohort Studies of Clinical Trials," PLOS Medicine, Public Library of Science, vol. 11(6), pages 1-22, June.
    3. Issa J. Dahabreh & Sarah E. Robertson & Lucia C. Petito & Miguel A. Hernán & Jon A. Steingrimsson, 2023. "Efficient and robust methods for causally interpretable meta‐analysis: Transporting inferences from multiple randomized trials to a target population," Biometrics, The International Biometric Society, vol. 79(2), pages 1057-1072, June.
    4. Williams, Martin J., 2019. "External validity and policy adaptation: From impact evaluation to policy design," PEGNet Policy Briefs 18/2019, PEGNet - Poverty Reduction, Equity and Growth Network, Kiel Institute for the World Economy (IfW Kiel).
    5. Manski, Charles F., 2013. "Public Policy in an Uncertain World: Analysis and Decisions," Economics Books, Harvard University Press, number 9780674066892, Spring.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2020. "Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis," Econometrica, Econometric Society, vol. 88(1), pages 265-296, January.
    2. Charles F. Manski & John V. Pepper, 2018. "How Do Right-to-Carry Laws Affect Crime Rates? Coping with Ambiguity Using Bounded-Variation Assumptions," The Review of Economics and Statistics, MIT Press, vol. 100(2), pages 232-244, May.
    3. Muller, Seán M., 2021. "The dangers of performative scientism as the alternative to anti-scientific policymaking: A critical, preliminary assessment of South Africa’s Covid-19 response and its consequences," World Development, Elsevier, vol. 140(C).
    4. Millimet, Daniel L. & Roy, Jayjit, 2015. "Multilateral environmental agreements and the WTO," Economics Letters, Elsevier, vol. 134(C), pages 20-23.
    5. Huhr, Scott & Wulczyn, Fred, 2022. "Do intensive in-home services prevent placement?: A case study of Youth Villages’ Intercept® program," Children and Youth Services Review, Elsevier, vol. 132(C).
    6. ILes, Richard, 2017. "Government Doctor Absenteeism And Its Effects On Consumer Demand In Rural North India," Working Papers 2018-9, School of Economic Sciences, Washington State University, revised 12 2018.
    7. Susan Athey & Raj Chetty & Guido Imbens, 2020. "Combining Experimental and Observational Data to Estimate Treatment Effects on Long Term Outcomes," Papers 2006.09676, arXiv.org.
    8. Charles F. Manski, 2020. "Towards Reasonable Patient Care Under Uncertainty," Contemporary Economic Policy, Western Economic Association International, vol. 38(2), pages 227-245, April.
    9. Raffaella Giacomini & Toru Kitagawa, 2021. "Robust Bayesian Inference for Set‐Identified Models," Econometrica, Econometric Society, vol. 89(4), pages 1519-1556, July.
    10. Charles F. Manski, 2018. "Reasonable patient care under uncertainty," Health Economics, John Wiley & Sons, Ltd., vol. 27(10), pages 1397-1421, October.
    11. Reiner Eichenberger & Rainer Hegselmann & David Savage & David Stadelmann & Benno Torgler, 2020. "Certified Corona-Immunity as a Resource and Strategy to Cope with Pandemic Costs," CREMA Working Paper Series 2020-03, Center for Research in Economics, Management and the Arts (CREMA).
    12. Ho Fai Chan & Nikita Ferguson & David A. Savage & David Stadelmann & Benno Torgler, 2020. "Is Science Able to Perform Under Pressure? Insights from COVID-19," CREMA Working Paper Series 2020-07, Center for Research in Economics, Management and the Arts (CREMA).
    13. Frimmel, Wolfgang & Halla, Martin & Winter-Ebmer, Rudolf, 2024. "How does parental divorce affect children’s long-term outcomes?," Journal of Public Economics, Elsevier, vol. 239(C).
    14. Eric Danan & Thibault Gajdos & Brian Hill & Jean-Marc Tallon, 2016. "Robust Social Decisions," American Economic Review, American Economic Association, vol. 106(9), pages 2407-2425, September.
    15. Fernando Hoces de la Guardia & Sean Grant & Edward Miguel, 2021. "A framework for open policy analysis," Science and Public Policy, Oxford University Press, vol. 48(2), pages 154-163.
    16. Guido W. Imbens, 2020. "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 1129-1179, December.
    17. Aizawa, T.;, 2019. "Reviewing the Existing Evidence of the Conditional Cash Transfer in India through the Partial Identification Approach," Health, Econometrics and Data Group (HEDG) Working Papers 19/24, HEDG, c/o Department of Economics, University of York.
    18. Xiaoyu Cheng, 2022. "Robust Data-Driven Decisions Under Model Uncertainty," Papers 2205.04573, arXiv.org.
    19. Kaushik Basu, 2014. "Randomisation, Causality and the Role of Reasoned Intuition," Oxford Development Studies, Taylor & Francis Journals, vol. 42(4), pages 455-472, December.
    20. Binswanger, Johannes & Oechslin, Manuel, 2020. "Better statistics, better economic policies?," European Economic Review, Elsevier, vol. 130(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:48:y:2024:i:3:p:427-460. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.