IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i5p436-471.html
   My bibliography  Save this article

Interaction of Theory and Practice to Assess External Validity

Author

Listed:
  • Laura C. Leviton
  • Mathew D. Trujillo

Abstract

Background: Variations in local context bedevil the assessment of external validity: the ability to generalize about effects of treatments. For evaluation, the challenges of assessing external validity are intimately tied to the translation and spread of evidence-based interventions. This makes external validity a question for decision makers, who need to determine whether to endorse, fund, or adopt interventions that were found to be effective and how to ensure high quality once they spread. Objective: To present the rationale for using theory to assess external validity and the value of more systematic interaction of theory and practice. Methods: We review advances in external validity, program theory, practitioner expertise, and local adaptation. Examples are provided for program theory, its adaptation to diverse contexts, and generalizing to contexts that have not yet been studied. The often critical role of practitioner experience is illustrated in these examples. Work is described that the Robert Wood Johnson Foundation is supporting to study treatment variation and context more systematically. Results: Researchers and developers generally see a limited range of contexts in which the intervention is implemented. Individual practitioners see a different and often a wider range of contexts, albeit not a systematic sample. Organized and taken together, however, practitioner experiences can inform external validity by challenging the developers and researchers to consider a wider range of contexts. Researchers have developed a variety of ways to adapt interventions in light of such challenges. Conclusions: In systematic programs of inquiry, as opposed to individual studies, the problems of context can be better addressed. Evaluators have advocated an interaction of theory and practice for many years, but the process can be made more systematic and useful. Systematic interaction can set priorities for assessment of external validity by examining the prevalence and importance of context features and treatment variations. Practitioner interaction with researchers and developers can assist in sharpening program theory, reducing uncertainty about treatment variations that are consistent or inconsistent with the theory, inductively ruling out the ones that are harmful or irrelevant, and helping set priorities for more rigorous study of context and treatment variation.

Suggested Citation

  • Laura C. Leviton & Mathew D. Trujillo, 2017. "Interaction of Theory and Practice to Assess External Validity," Evaluation Review, , vol. 41(5), pages 436-471, October.
    • Handle: RePEc:sae:evarev:v:41:y:2017:i:5:p:436-471
    DOI: 10.1177/0193841X15625289
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X15625289
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X15625289?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Glasgow, R.E. & Vogt, T.M. & Boles, S.M., 1999. "Evaluating the public health impact of health promotion interventions: The RE-AIM framework," American Journal of Public Health, American Public Health Association, vol. 89(9), pages 1322-1327.
    2. Foshee, V.A. & Bauman, K.E. & Ennett, S.T. & Linder, G.F. & Benefield, T. & Suchindran, C., 2004. "Assessing the Long-Term Effects of the Safe Dates Program and a Booster in Preventing and Reducing Adolescent Dating Violence Victimization and Perpetration," American Journal of Public Health, American Public Health Association, vol. 94(4), pages 619-624.
    3. Andreas Mraz Esposito & Rebecca Coughlin & Steven Malick & Emily Sama-Miller & Patricia Del Grosso & Rebecca Kleinman & Diane Paulsell, "undated". "Assessing the Research on Home Visiting Program Models Implemented in Tribal Communities, Part 1: Evidence of Effectiveness," Mathematica Policy Research Reports f7687fac6ba64b4596b382672, Mathematica Policy Research.
    4. Chen, Huey T., 2010. "The bottom-up approach to integrative validity: A new perspective for program evaluation," Evaluation and Program Planning, Elsevier, vol. 33(3), pages 205-214, August.
    5. Schuh, Russell G. & Leviton, Laura C., 2006. "A framework to assess the development and capacity of non-profit agencies," Evaluation and Program Planning, Elsevier, vol. 29(2), pages 171-179, May.
    6. Patricia Del Grosso & Rebecca Kleinman & Andrea Mraz Esposito & Emily Sama Martin & Diane Paulsell, "undated". "Assessing the Evidence of Effectiveness of Home Visiting Program Models Implemented in Tribal Communities," Mathematica Policy Research Reports 6b7a19bd424149adabdfd4590, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Naylor, Mary D. & Hirschman, Karen B. & Toles, Mark P. & Jarrín, Olga F. & Shaid, Elizabeth & Pauly, Mark V., 2018. "Adaptations of the evidence-based Transitional Care Model in the U.S," Social Science & Medicine, Elsevier, vol. 213(C), pages 28-36.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Green, Beth L. & Tarte, Jerod M. & Harrison, Paige M. & Nygren, Margaret & Sanders, Mary Beth, 2014. "Results from a randomized trial of the Healthy Families Oregon accredited statewide program: Early program impacts on parenting," Children and Youth Services Review, Elsevier, vol. 44(C), pages 288-298.
    2. Kimberly Boller, 2012. "Evidence for the Role of Home Visiting in Child Maltreatment Prevention," Mathematica Policy Research Reports 32c88b1fccd0493b970883942, Mathematica Policy Research.
    3. Debra A. Strong & Sarah A. Avellar & Caroline Massad Francis & Megan Hague Angus & Andrea Mraz Esposito, "undated". "Serving Child Welfare Families with Substance Abuse Issues: Grantees' Use of Evidence-Based Practices and the Extent of Evidence," Mathematica Policy Research Reports db654cbf07ec49d5b0d8a5a11, Mathematica Policy Research.
    4. Nikki Aikens & Yange Xue & Eileen Bandel & Pia Caronongan & Cheri A. Vogel & Kimberly Boller, 2015. "Early Head Start Home Visits and Classrooms: Stability, Predictors, and Thresholds of Quality," Mathematica Policy Research Reports 9fd24acaa7de4a9b8edec7756, Mathematica Policy Research.
    5. repec:mpr:mprres:7551 is not listed on IDEAS
    6. Jungyoon Kim & Valerie Pacino & Hongmei Wang & April Recher & Isha Jain & Vaibhavi Mone & Jihyun Ma & Mary Jo Spurgin & Daniel Jeffrey & Stephen Mohring & Jane Potter, 2021. "System Redesign: The Value of a Primary Care Liaison Model to Address Unmet Social Needs among Older Primary Care Patients," IJERPH, MDPI, vol. 18(21), pages 1-11, October.
    7. Gonot-Schoupinsky, Freda N. & Garip, Gulcan, 2019. "A flexible framework for planning and evaluating early-stage health interventions: FRAME-IT," Evaluation and Program Planning, Elsevier, vol. 77(C).
    8. Chelsea M. Cooper & Mary Drake & Justine A. Kavle & Joyce Nyoni & Ruth Lemwayi & Lemmy Mabuga & Anne Pfitzer & Mary Makungu & Elizabeth Massawe & John George, 2021. "Implementing a Novel Facility-Community Intervention for Strengthening Integration of Infant Nutrition and Family Planning in Mara and Kagera, Tanzania," IJERPH, MDPI, vol. 18(8), pages 1-21, April.
    9. Francisco Javier de la Garza Iga & Marinés Mejía Alvarez & Joshua D Cockroft & Julia Rabin & Ana Cordón & Dina Maria Elias Rodas & Maria del Pilar Grazioso & Maria Espinola & Christine O’Dea & Ch, 2023. "Using the project ECHO™ model to teach mental health topics in rural Guatemala: An implementation science-guided evaluation," International Journal of Social Psychiatry, , vol. 69(8), pages 2031-2041, December.
    10. Eugenio Zucchelli & Andrew M Jones & Nigel Rice, 2012. "The evaluation of health policies through dynamic microsimulation methods," International Journal of Microsimulation, International Microsimulation Association, vol. 5(1), pages 2-20.
    11. Gabrielle Scronce & Wanqing Zhang & Matthew Lee Smith & Vicki Stemmons Mercer, 2020. "Characteristics Associated with Improved Physical Performance among Community-Dwelling Older Adults in a Community-Based Falls Prevention Program," IJERPH, MDPI, vol. 17(7), pages 1-12, April.
    12. Saria Hassan & Alexis Cooke & Haneefa Saleem & Dorothy Mushi & Jessie Mbwambo & Barrot H. Lambdin, 2019. "Evaluating the Integrated Methadone and Anti-Retroviral Therapy Strategy in Tanzania Using the RE-AIM Framework," IJERPH, MDPI, vol. 16(5), pages 1-15, February.
    13. Julie Cowie & Eileen Calveley & Gillian Bowers & John Bowers, 2018. "Evaluation of a Digital Consultation and Self-Care Advice Tool in Primary Care: A Multi-Methods Study," IJERPH, MDPI, vol. 15(5), pages 1-23, May.
    14. Stanley, Nicky & Ellis, Jane & Farrelly, Nicola & Hollinghurst, Sandra & Downe, Soo, 2015. "Preventing domestic abuse for children and young people: A review of school-based interventions," Children and Youth Services Review, Elsevier, vol. 59(C), pages 120-131.
    15. Archibald, Thomas, 2015. "“They Just Know”: The epistemological politics of “evidence-based” non-formal education," Evaluation and Program Planning, Elsevier, vol. 48(C), pages 137-148.
    16. Melissa D. Olfert & Rebecca L. Hagedorn & Makenzie L. Barr & Oluremi A. Famodu & Jessica M. Rubino & Jade A. White, 2018. "eB4CAST: An Evidence-Based Tool to Promote Dissemination and Implementation in Community-Based, Public Health Research," IJERPH, MDPI, vol. 15(10), pages 1-13, September.
    17. Urban, Jennifer Brown & Hargraves, Monica & Trochim, William M., 2014. "Evolutionary Evaluation: Implications for evaluators, researchers, practitioners, funders and the evidence-based program mandate," Evaluation and Program Planning, Elsevier, vol. 45(C), pages 127-139.
    18. Sebastian-Ion Ceptureanu & Eduard-Gabriel Ceptureanu & Mihai Cristian Orzan & Irinel Marin, 2017. "Toward a Romanian NPOs Sustainability Model: Determinants of Sustainability," Sustainability, MDPI, vol. 9(6), pages 1-26, June.
    19. Michelle Nichols & Ronald Teufel & Sarah Miller & Mohan Madisetti & Christine San Giovanni & Katherine Chike-Harris & Lacy Jones & Margaret Prentice & Kenneth Ruggiero & Teresa Kelechi, 2020. "Managing Asthma and Obesity Related Symptoms (MATADORS): An mHealth Intervention to Facilitate Symptom Self-Management among Youth," IJERPH, MDPI, vol. 17(21), pages 1-15, October.
    20. Natalie Bradford & Shirley Chambers & Adrienne Hudson & Jacqui Jauncey‐Cooke & Robyn Penny & Carol Windsor & Patsy Yates, 2019. "Evaluation frameworks in health services: An integrative review of use, attributes and elements," Journal of Clinical Nursing, John Wiley & Sons, vol. 28(13-14), pages 2486-2498, July.
    21. Seonad K. Madden & Helen Skouteris & Cate Bailey & Andrew P. Hills & Kiran D. K. Ahuja & Briony Hill, 2020. "Women in the Workplace: Promoting Healthy Lifestyles and Mitigating Weight Gain during the Preconception, Pregnancy, and Postpartum Periods," IJERPH, MDPI, vol. 17(3), pages 1-7, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:5:p:436-471. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.