IDEAS home Printed from https://ideas.repec.org/a/eee/epplan/v91y2022ics0149718922000106.html
   My bibliography  Save this article

What does it mean for an evaluation to be ‘valid’? A critical synthesis of evaluation literature

Author

Listed:
  • Downes, Jenni
  • Gullickson, Amy M.

Abstract

Validity is both a critical foundation for, and widely contested concept in the field of program evaluation. Given the wide and divergent treatment of the term, this article sought to enhance the field’s understanding and application of validity by systematically identifying, collating and integrating existing conceptions of validity. A critical interpretive synthesis approach was taken to combine the best elements of systematic literature searching with integrative review. We found that conceptions of validity could be roughly collated into three categories pertaining to the evaluation research, evaluative judgement and reporting, and beyond the evaluation proper. The result was a wide-ranging map of validity concepts against the various stages of the evaluation process. This made clear several gaps in the validity thinking, resulting in the development of comprehensive but not yet complete framework that weaves together related aspects identified into a cohesive whole. It is our hope that this initial iteration of a cohesive validity framework will be the starting point for a cooperative effort to improve professional evaluation practice.

Suggested Citation

  • Downes, Jenni & Gullickson, Amy M., 2022. "What does it mean for an evaluation to be ‘valid’? A critical synthesis of evaluation literature," Evaluation and Program Planning, Elsevier, vol. 91(C).
    • Handle: RePEc:eee:epplan:v:91:y:2022:i:c:s0149718922000106
    DOI: 10.1016/j.evalprogplan.2022.102056
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0149718922000106
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.evalprogplan.2022.102056?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Arbour, Ghislain, 2020. "Teaching programme evaluation: A problem of knowledge," Evaluation and Program Planning, Elsevier, vol. 83(C).
    2. Benjamin Saunders & Julius Sim & Tom Kingstone & Shula Baker & Jackie Waterfield & Bernadette Bartlam & Heather Burroughs & Clare Jinks, 2018. "Saturation in qualitative research: exploring its conceptualization and operationalization," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(4), pages 1893-1907, July.
    3. Patton, Michael Quinn, 1989. "A context and boundaries for a theory-driven approach to validity," Evaluation and Program Planning, Elsevier, vol. 12(4), pages 375-377, January.
    4. Palumbo, Dennis J. & Oliverio, Annamarie, 1989. "Implementation theory and the theory-driven approach to validity," Evaluation and Program Planning, Elsevier, vol. 12(4), pages 337-344, January.
    5. Chen, Huey-Tsyh & Rossi, Peter H., 1987. "The theory-driven approach to validity," Evaluation and Program Planning, Elsevier, vol. 10(1), pages 95-103, January.
    6. Chen, Huey T., 2010. "The bottom-up approach to integrative validity: A new perspective for program evaluation," Evaluation and Program Planning, Elsevier, vol. 33(3), pages 205-214, August.
    7. Garaway, G. B., 1997. "Evaluation, validity, and values," Evaluation and Program Planning, Elsevier, vol. 20(1), pages 1-5, February.
    8. Gullickson, Amy M. & King, Jean A. & LaVelle, John M. & Clinton, Janet M., 2019. "The current state of evaluator education: A situation analysis and call to action," Evaluation and Program Planning, Elsevier, vol. 75(C), pages 20-30.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jennifer Tartaglia & Michelle McIntosh & Jonine Jancey & Jane Scott & Andrea Begley, 2021. "Exploring Feeding Practices and Food Literacy in Parents with Young Children from Disadvantaged Areas," IJERPH, MDPI, vol. 18(4), pages 1-18, February.
    2. Şahika Simsek-Cetinkaya & Simge Evrenol Ocal, 2023. "“Psychological Injuries Are Not Visible†: Experiences and Perceptions of Midwives and Nurses about Domestic Violence during Pregnancy," Clinical Nursing Research, , vol. 32(8), pages 1115-1123, November.
    3. Archibald, Thomas, 2015. "“They Just Know”: The epistemological politics of “evidence-based” non-formal education," Evaluation and Program Planning, Elsevier, vol. 48(C), pages 137-148.
    4. Silvia Scaramuzzi & Sara Gabellini & Giovanni Belletti & Andrea Marescotti, 2021. "Agrobiodiversity-Oriented Food Systems between Public Policies and Private Action: A Socio-Ecological Model for Sustainable Territorial Development," Sustainability, MDPI, vol. 13(21), pages 1-32, November.
    5. Urban, Jennifer Brown & Hargraves, Monica & Trochim, William M., 2014. "Evolutionary Evaluation: Implications for evaluators, researchers, practitioners, funders and the evidence-based program mandate," Evaluation and Program Planning, Elsevier, vol. 45(C), pages 127-139.
    6. Tongyu Meng & Jamie Newth & Christine Woods, 2022. "Ethical Sensemaking in Impact Investing: Reasons and Motives in the Chinese Renewable Energy Sector," Journal of Business Ethics, Springer, vol. 179(4), pages 1091-1117, September.
    7. Annette Peart & Virginia Lewis & Chris Barton & Grant Russell, 2020. "Healthcare professionals providing care coordination to people living with multimorbidity: An interpretative phenomenological analysis," Journal of Clinical Nursing, John Wiley & Sons, vol. 29(13-14), pages 2317-2328, July.
    8. Soo-Yong Shin & Eun-Ju Lim, 2021. "Clinical Work and Life of Mid-Career Male Nurses: A Qualitative Study," IJERPH, MDPI, vol. 18(12), pages 1-10, June.
    9. Fatoumata Fofana & Pat Bazeley & Antoine Regnault, 2020. "Applying a mixed methods design to test saturation for qualitative data in health outcomes research," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-12, June.
    10. Shern, David L. & Trochim, William M. K. & LaComb, Christina A., 1995. "The use of concept mapping for assessing fidelity of model transfer: An example from psychiatric rehabilitation," Evaluation and Program Planning, Elsevier, vol. 18(2), pages 143-153.
    11. Thomas Dax & Oliver Tamme, 2023. "Attractive Landscape Features as Drivers for Sustainable Mountain Tourism Experiences," Tourism and Hospitality, MDPI, vol. 4(3), pages 1-16, June.
    12. Dimitrios Gouglas & Kendall Hoyt & Elizabeth Peacocke & Aristidis Kaloudis & Trygve Ottersen & John-Arne Røttingen, 2019. "Setting Strategic Objectives for the Coalition for Epidemic Preparedness Innovations: An Exploratory Decision Analysis Process," Service Science, INFORMS, vol. 49(6), pages 430-446, November.
    13. Matthys, Marie-Luise & Acharya, Sushant & Khatri, Sanjaya, 2021. "“Before cardamom, we used to face hardship”: Analyzing agricultural commercialization effects in Nepal through a local concept of the Good Life," World Development, Elsevier, vol. 141(C).
    14. Radomska Joanna & Hajdas Monika & Wołczek Przemysław & Glinka Beata, 2023. "Wide open? Creative industries and open strategizing challenges," International Journal of Management and Economics, Warsaw School of Economics, Collegium of World Economy, vol. 59(2), pages 117-136, June.
    15. Stephanie W. Lee & Jing Xu & Tai-Ming Wut & Yui-Yip Lau & Joseph H. L. Chan & Tin-Shing Liu & Louis W. Y. Mok & Jason K. Y. Chan, 2024. "Aging in Place in Hong Kong and Its Implications for Sustainable Development: A Qualitative Study Exploring the Needs, Beliefs, Behaviors, and Well-Being of Older Adults through Self-Determination The," Sustainability, MDPI, vol. 16(8), pages 1-18, April.
    16. Shailaja Tetali & Sureshkumar Kamalakannan & Shilpa Sadanand & Melissa Glenda Lewis & Sara Varughese & Annie Hans & G. V. S. Murthy, 2022. "Evaluation of the Impact of the First Wave of COVID-19 and Associated Lockdown Restrictions on Persons with Disabilities in 14 States of India," IJERPH, MDPI, vol. 19(18), pages 1-14, September.
    17. Benson, Paul R. & Fisher, Gene A. & Diana, Augusto & Simon, Lorna & Gamache, Gail & Tessler, Richard C. & McDermeit, Melissa, 1996. "A state network of family support services: The massachusetts family support demonstration project," Evaluation and Program Planning, Elsevier, vol. 19(1), pages 27-39, February.
    18. John R. Blakeman & Wendy M. Woith & Kim S. Astroth & Sheryl H. Jenkins & Stephen J. Stapleton, 2020. "A qualitative exploration of prodromal myocardial infarction fatigue experienced by women," Journal of Clinical Nursing, John Wiley & Sons, vol. 29(19-20), pages 3882-3895, October.
    19. Fahlstedt, Oskar & Temeljotov-Salaj, Alenka & Lohne, Jardar & Bohne, Rolf André, 2022. "Holistic assessment of carbon abatement strategies in building refurbishment literature — A scoping review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    20. Kent, Douglas R. & Donaldson, Stewart I. & Wyrick, Phelan A. & Smith, Peggy J., 2000. "Evaluating criminal justice programs designed to reduce crime by targeting repeat gang offenders," Evaluation and Program Planning, Elsevier, vol. 23(1), pages 115-124, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:epplan:v:91:y:2022:i:c:s0149718922000106. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/evalprogplan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.