IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i4p283-325.html
   My bibliography  Save this article

External Validity: The Next Step for Systematic Reviews?

Author

Listed:
  • Sarah A. Avellar
  • Jaime Thomas
  • Rebecca Kleinman
  • Emily Sama-Miller
  • Sara E. Woodruff
  • Rebecca Coughlin
  • T’Pring R. Westbrook

Abstract

Background: Systematic reviews—which identify, assess, and summarize existing research—are usually designed to determine whether research shows that an intervention has evidence of effectiveness, rather than whether an intervention will work under different circumstances. The reviews typically focus on the internal validity of the research and do not consistently incorporate information on external validity into their conclusions. Objectives: In this article, we focus on how systematic reviews address external validity. Methods: We conducted a brief scan of 19 systematic reviews and a more in-depth examination of information presented in a systematic review of home visiting research. Results: We found that many reviews do not provide information on generalizability, such as statistical representativeness, but focus on factors likely to increase heterogeneity (e.g., numbers of studies or settings) and report on context. The latter may help users decide whether the research characteristics—such as sample demographics or settings—are similar to their own. However, we found that differences in reporting, such as which variables are included and how they are measured, make it difficult to summarize across studies or make basic determinations of sample characteristics, such as whether the majority of a sample was unemployed or married. Conclusion: Evaluation research and systematic reviews would benefit from reporting guidelines for external validity to ensure that key information is reported across studies.

Suggested Citation

  • Sarah A. Avellar & Jaime Thomas & Rebecca Kleinman & Emily Sama-Miller & Sara E. Woodruff & Rebecca Coughlin & T’Pring R. Westbrook, 2017. "External Validity: The Next Step for Systematic Reviews?," Evaluation Review, , vol. 41(4), pages 283-325, August.
  • Handle: RePEc:sae:evarev:v:41:y:2017:i:4:p:283-325
    DOI: 10.1177/0193841X16665199
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X16665199
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0193841X16665199?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    2. Robert B. Olsen & Larry L. Orr & Stephen H. Bell & Elizabeth A. Stuart, 2013. "External Validity in Policy Evaluations That Choose Sites Purposively," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 32(1), pages 107-121, January.
    3. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    4. Steckler, A. & McLeroy, K.R., 2008. "The importance of external validity," American Journal of Public Health, American Public Health Association, vol. 98(1), pages 9-10.
    5. repec:mpr:mprres:6574 is not listed on IDEAS
    6. Chen, Huey T., 2010. "The bottom-up approach to integrative validity: A new perspective for program evaluation," Evaluation and Program Planning, Elsevier, vol. 33(3), pages 205-214, August.
    7. Paulsell, D. & Del Grosso, P. & Supplee, L., 2014. "Supporting replication and scale-up of evidence-based home visiting programs: Assessing the implementation knowledge base," American Journal of Public Health, American Public Health Association, vol. 104(9), pages 1624-1632.
    8. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    9. Changyu Shen & Jaesik Jeong & Xiaochun Li & Peng-Sheng Chen & Alfred Buxton, 2013. "Treatment Benefit and Treatment Harm Rate to Characterize Heterogeneity in Treatment Effect," Biometrics, The International Biometric Society, vol. 69(3), pages 724-731, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christian Leuz, 2018. "Evidence-based policymaking: promise, challenges and opportunities for accounting and financial markets research," Accounting and Business Research, Taylor & Francis Journals, vol. 48(5), pages 582-608, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jacob Alex Klerman, 2017. "Editor in Chief’s Comment: External Validity in Systematic Reviews," Evaluation Review, , vol. 41(5), pages 391-402, October.
    2. repec:mpr:mprres:8128 is not listed on IDEAS
    3. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    4. Sharples, Linda D., 2018. "The role of statistics in the era of big data: Electronic health records for healthcare research," Statistics & Probability Letters, Elsevier, vol. 136(C), pages 105-110.
    5. Elizabeth A. Stuart & Anna Rhodes, 2017. "Generalizing Treatment Effect Estimates From Sample to Population: A Case Study in the Difficulties of Finding Sufficient Data," Evaluation Review, , vol. 41(4), pages 357-388, August.
    6. Elizabeth Tipton, 2021. "Beyond generalization of the ATE: Designing randomized trials to understand treatment effect heterogeneity," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 504-521, April.
    7. Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2021. "From Local to Global: External Validity in a Fertility Natural Experiment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 217-243, January.
    8. Peter Z. Schochet, "undated". "Statistical Theory for the RCT-YES Software: Design-Based Causal Inference for RCTs," Mathematica Policy Research Reports a0c005c003c242308a92c02dc, Mathematica Policy Research.
    9. Xinkun Nie & Guido Imbens & Stefan Wager, 2021. "Covariate Balancing Sensitivity Analysis for Extrapolating Randomized Trials across Locations," Papers 2112.04723, arXiv.org.
    10. Wendy Chan, 2018. "Applications of Small Area Estimation to Generalization With Subclassification by Propensity Scores," Journal of Educational and Behavioral Statistics, , vol. 43(2), pages 182-224, April.
    11. Elizabeth Tipton & Kelly Hallberg & Larry V. Hedges & Wendy Chan, 2017. "Implications of Small Samples for Generalization: Adjustments and Rules of Thumb," Evaluation Review, , vol. 41(5), pages 472-505, October.
    12. Diane Paulsell & Jaime Thomas & Shannon Monahan & Neil S. Seftor, 2017. "A Trusted Source of Information," Evaluation Review, , vol. 41(1), pages 50-77, February.
    13. Esterling, Kevin & Brady, David & Schwitzgebel, Eric, 2021. "The Necessity of Construct and External Validity for Generalized Causal Claims," OSF Preprints 2s8w5, Center for Open Science.
    14. Chen, Huey T. & Yip, Fuyuen & Lavonas, Eric J. & Iqbal, Shahed & Turner, Nannette & Cobb, Bobby & Garbe, Paul, 2014. "Using the exhibited generalization approach to evaluate a carbon monoxide alarm ordinance," Evaluation and Program Planning, Elsevier, vol. 47(C), pages 35-44.
    15. Andrews, Isaiah & Oster, Emily, 2019. "A simple approximation for evaluating external validity bias," Economics Letters, Elsevier, vol. 178(C), pages 58-62.
    16. Elizabeth Tipton & Laura R. Peck, 2017. "A Design-Based Approach to Improve External Validity in Welfare Policy Evaluations," Evaluation Review, , vol. 41(4), pages 326-356, August.
    17. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    18. Elizabeth Tipton, 2014. "How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations," Journal of Educational and Behavioral Statistics, , vol. 39(6), pages 478-501, December.
    19. Fabian Kosse & Thomas Deckers & Pia Pinger & Hannah Schildberg-Hörisch & Armin Falk, 2020. "The Formation of Prosociality: Causal Evidence on the Role of Social Environment," Journal of Political Economy, University of Chicago Press, vol. 128(2), pages 434-467.
    20. Ángel Enrique & Juana Bretón-López & Guadalupe Molinari & Rosa M. Baños & Cristina Botella, 2018. "Efficacy of an adaptation of the Best Possible Self intervention implemented through positive technology: a randomized control trial," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 13(3), pages 671-689, September.
    21. Gerben ter Riet & Paula Chesley & Alan G Gross & Lara Siebeling & Patrick Muggensturm & Nadine Heller & Martin Umbehr & Daniela Vollenweider & Tsung Yu & Elie A Akl & Lizzy Brewster & Olaf M Dekkers &, 2013. "All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies," PLOS ONE, Public Library of Science, vol. 8(11), pages 1-6, November.

    More about this item

    Keywords

    external validity; systematic reviews;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:4:p:283-325. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.