IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i5p436-471.html
   My bibliography  Save this article

Interaction of Theory and Practice to Assess External Validity

Author

Listed:
  • Laura C. Leviton
  • Mathew D. Trujillo

Abstract

Background: Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread. Objective: To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice. Methods: We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically. Results: Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges. Conclusions: In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

Suggested Citation

  • Laura C. Leviton & Mathew D. Trujillo, 2017. "Interaction of Theory and Practice to Assess External Validity," Evaluation Review, , vol. 41(5), pages 436-471, October.
    • Handle: RePEc:sae:evarev:v:41:y:2017:i:5:p:436-471
    DOI: 10.1177/0193841X15625289
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X15625289
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X15625289?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Glasgow, R.E. & Vogt, T.M. & Boles, S.M., 1999. "Evaluating the public health impact of health promotion interventions: The RE-AIM framework," American Journal of Public Health, American Public Health Association, vol. 89(9), pages 1322-1327.
    2. Foshee, V.A. & Bauman, K.E. & Ennett, S.T. & Linder, G.F. & Benefield, T. & Suchindran, C., 2004. "Assessing the Long-Term Effects of the Safe Dates Program and a Booster in Preventing and Reducing Adolescent Dating Violence Victimization and Perpetration," American Journal of Public Health, American Public Health Association, vol. 94(4), pages 619-624.
    3. Andreas Mraz Esposito & Rebecca Coughlin & Steven Malick & Emily Sama-Miller & Patricia Del Grosso & Rebecca Kleinman & Diane Paulsell, "undated". "Assessing the Research on Home Visiting Program Models Implemented in Tribal Communities, Part 1: Evidence of Effectiveness," Mathematica Policy Research Reports f7687fac6ba64b4596b382672, Mathematica Policy Research.
    4. Chen, Huey T., 2010. "The bottom-up approach to integrative validity: A new perspective for program evaluation," Evaluation and Program Planning, Elsevier, vol. 33(3), pages 205-214, August.
    5. Schuh, Russell G. & Leviton, Laura C., 2006. "A framework to assess the development and capacity of non-profit agencies," Evaluation and Program Planning, Elsevier, vol. 29(2), pages 171-179, May.
    6. Patricia Del Grosso & Rebecca Kleinman & Andrea Mraz Esposito & Emily Sama Martin & Diane Paulsell, "undated". "Assessing the Evidence of Effectiveness of Home Visiting Program Models Implemented in Tribal Communities," Mathematica Policy Research Reports 6b7a19bd424149adabdfd4590, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Naylor, Mary D. & Hirschman, Karen B. & Toles, Mark P. & Jarrín, Olga F. & Shaid, Elizabeth & Pauly, Mark V., 2018. "Adaptations of the evidence-based Transitional Care Model in the U.S," Social Science & Medicine, Elsevier, vol. 213(C), pages 28-36.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kimberly Boller, 2012. "Evidence for the Role of Home Visiting in Child Maltreatment Prevention," Mathematica Policy Research Reports 32c88b1fccd0493b970883942, Mathematica Policy Research.
    2. Nikki Aikens & Yange Xue & Eileen Bandel & Pia Caronongan & Cheri A. Vogel & Kimberly Boller, 2015. "Early Head Start Home Visits and Classrooms: Stability, Predictors, and Thresholds of Quality," Mathematica Policy Research Reports 9fd24acaa7de4a9b8edec7756, Mathematica Policy Research.
    3. Green, Beth L. & Tarte, Jerod M. & Harrison, Paige M. & Nygren, Margaret & Sanders, Mary Beth, 2014. "Results from a randomized trial of the Healthy Families Oregon accredited statewide program: Early program impacts on parenting," Children and Youth Services Review, Elsevier, vol. 44(C), pages 288-298.
    4. Debra A. Strong & Sarah A. Avellar & Caroline Massad Francis & Megan Hague Angus & Andrea Mraz Esposito, "undated". "Serving Child Welfare Families with Substance Abuse Issues: Grantees' Use of Evidence-Based Practices and the Extent of Evidence," Mathematica Policy Research Reports db654cbf07ec49d5b0d8a5a11, Mathematica Policy Research.
    5. repec:mpr:mprres:7551 is not listed on IDEAS
    6. Chelsea M. Cooper & Mary Drake & Justine A. Kavle & Joyce Nyoni & Ruth Lemwayi & Lemmy Mabuga & Anne Pfitzer & Mary Makungu & Elizabeth Massawe & John George, 2021. "Implementing a Novel Facility-Community Intervention for Strengthening Integration of Infant Nutrition and Family Planning in Mara and Kagera, Tanzania," IJERPH, MDPI, vol. 18(8), pages 1-21, April.
    7. Francisco Javier de la Garza Iga & Marinés Mejía Alvarez & Joshua D Cockroft & Julia Rabin & Ana Cordón & Dina Maria Elias Rodas & Maria del Pilar Grazioso & Maria Espinola & Christine O’Dea & Ch, 2023. "Using the project ECHO™ model to teach mental health topics in rural Guatemala: An implementation science-guided evaluation," International Journal of Social Psychiatry, , vol. 69(8), pages 2031-2041, December.
    8. Stanley, Nicky & Ellis, Jane & Farrelly, Nicola & Hollinghurst, Sandra & Downe, Soo, 2015. "Preventing domestic abuse for children and young people: A review of school-based interventions," Children and Youth Services Review, Elsevier, vol. 59(C), pages 120-131.
    9. Archibald, Thomas, 2015. "“They Just Know”: The epistemological politics of “evidence-based” non-formal education," Evaluation and Program Planning, Elsevier, vol. 48(C), pages 137-148.
    10. Urban, Jennifer Brown & Hargraves, Monica & Trochim, William M., 2014. "Evolutionary Evaluation: Implications for evaluators, researchers, practitioners, funders and the evidence-based program mandate," Evaluation and Program Planning, Elsevier, vol. 45(C), pages 127-139.
    11. Sebastian-Ion Ceptureanu & Eduard-Gabriel Ceptureanu & Mihai Cristian Orzan & Irinel Marin, 2017. "Toward a Romanian NPOs Sustainability Model: Determinants of Sustainability," Sustainability, MDPI, vol. 9(6), pages 1-26, June.
    12. Michelle Nichols & Ronald Teufel & Sarah Miller & Mohan Madisetti & Christine San Giovanni & Katherine Chike-Harris & Lacy Jones & Margaret Prentice & Kenneth Ruggiero & Teresa Kelechi, 2020. "Managing Asthma and Obesity Related Symptoms (MATADORS): An mHealth Intervention to Facilitate Symptom Self-Management among Youth," IJERPH, MDPI, vol. 17(21), pages 1-15, October.
    13. Dominic Richardson & UNICEF Office of Research - Innocenti, 2018. "Key Findings on Families, Family Policy and the Sustainable Development Goals: Synthesis Report," Papers inorer948, Innocenti Research Report.
    14. Sylvie Pires & Hélène Denizot & Abdel Halim Boudoukha & Julie Mennuti & Cécile Miele & Catherine Potard & Gaëlle Riquoir & Pierre-Michel Llorca & Valentin Flaudias & Laurent Gerbaud, 2023. "Selflife : A Life Skills Development Tool to Prevent Sexual Violence among Healthcare Students," IJERPH, MDPI, vol. 20(6), pages 1-10, March.
    15. Floor A. van den Brand & Tessa Magnée & Lotte de Haan-Bouma & Cas Barendregt & Niels H. Chavannes & Onno C. P. van Schayck & Gera E. Nagelhout, 2019. "Implementation of Financial Incentives for Successful Smoking Cessation in Real-Life Company Settings: A Qualitative Needs Assessment among Employers," IJERPH, MDPI, vol. 16(24), pages 1-16, December.
    16. Currie, Melissa & King, Gillian & Rosenbaum, Peter & Law, Mary & Kertoy, Marilyn & Specht, Jacqueline, 2005. "A model of impacts of research partnerships in health and social services," Evaluation and Program Planning, Elsevier, vol. 28(4), pages 400-412, November.
    17. Guzmán, Azucena & Orellana, Katharine & Ortega, Mariana López & Robledo, Luis Miguel Gutiérrez & Castro, Sara Torres, 2024. "Introducing a multicomponent staff training intervention to reduce antipsychotic medication: Care home management pre and post intervention views of systemic impact, and preliminary RE-AIM evaluation," Evaluation and Program Planning, Elsevier, vol. 103(C).
    18. Petra Boersma & Julia C M van Weert & Berno van Meijel & Rose‐Marie Dröes, 2017. "Implementation of the Veder contact method in daily nursing home care for people with dementia: a process analysis according to the RE‐AIM framework," Journal of Clinical Nursing, John Wiley & Sons, vol. 26(3-4), pages 436-455, February.
    19. Marsha L. Brierley & Lindsey R. Smith & Angel M. Chater & Daniel P. Bailey, 2022. "A-REST (Activity to Reduce Excessive Sitting Time): A Feasibility Trial to Reduce Prolonged Sitting in Police Staff," IJERPH, MDPI, vol. 19(15), pages 1-27, July.
    20. Pearl Anne Ante-Testard & Francois Rerolle & Anna T. Nguyen & Sania Ashraf & Sarker Masud Parvez & Abu Mohammed Naser & Tarik Benmarhnia & Mahbubur Rahman & Stephen P. Luby & Jade Benjamin-Chung & Ben, 2024. "WASH interventions and child diarrhea at the interface of climate and socioeconomic position in Bangladesh," Nature Communications, Nature, vol. 15(1), pages 1-13, December.
    21. Katrien De Cocker & Greet Cardon & Jason A. Bennie & Tracy Kolbe-Alexander & Femke De Meester & Corneel Vandelanotte, 2018. "From Evidence-Based Research to Practice-Based Evidence: Disseminating a Web-Based Computer-Tailored Workplace Sitting Intervention through a Health Promotion Organisation," IJERPH, MDPI, vol. 15(5), pages 1-11, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:5:p:436-471. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.