IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v36y2012i5p375-401.html
   My bibliography  Save this article

When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model

Author

Listed:
  • Diana Epstein
  • Jacob Alex Klerman

Abstract

Background: Recent reviews suggest that many plausible programs are found to have at best small impacts not commensurate with their cost, and often have no detectable positive impacts at all. Even programs with initial rigorous impact evaluation (RIE) that show them to be effective often fail a second test with an expanded population or at multiple sites. Objective: This article argues that more rapid movement to RIE is a partial cause of the low success rate of RIE and proposes a constructive response: process evaluations that compare program intermediate outcomes—in the treatment group, during the operation of the program—against a more falsifiable extension of the conventional logic model. Conclusion: Our examples suggest that such process evaluations would allow funders to deem many programs unlikely to show impacts and therefore not ready for random assignment evaluation—without the high cost and long time lines of an RIE. The article then develops the broader implications of such a process analysis step for broader evaluation strategy.

Suggested Citation

  • Diana Epstein & Jacob Alex Klerman, 2012. "When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model," Evaluation Review, , vol. 36(5), pages 375-401, October.
    • Handle: RePEc:sae:evarev:v:36:y:2012:i:5:p:375-401
    DOI: 10.1177/0193841X12474275
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X12474275
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X12474275?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. McLaughlin, John A. & Jordan, Gretchen B., 1999. "Logic models: a tool for telling your programs performance story," Evaluation and Program Planning, Elsevier, vol. 22(1), pages 65-72.
    2. repec:mpr:mprres:4971 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Margaret Dalziel, 2018. "Why are there (almost) no randomised controlled trial-based evaluations of business support programmes?," Palgrave Communications, Palgrave Macmillan, vol. 4(1), pages 1-9, December.
    2. Thierry Bastelaer & Benjamin Woodman & Minki Chatterji & David Long, 2016. "Saving for delivery through technology: an inquiry into an electronic commitment savings product for maternal healthcare in Kenya," International Journal of Health Planning and Management, Wiley Blackwell, vol. 31(4), pages 254-272, October.
    3. Ofek, Yuval, 2017. "Evaluating social exclusion interventions in university-community partnerships," Evaluation and Program Planning, Elsevier, vol. 60(C), pages 46-55.
    4. Gates, Lauren B. & Pearlmutter, Sue & Keenan, Kat & Divver, Caitlin & Gorroochurn, Prakash, 2018. "Career readiness programming for youth in foster care," Children and Youth Services Review, Elsevier, vol. 89(C), pages 152-164.
    5. Elisa Ricciuti & Francesca Calò, 2018. "Are foundations assessing their impact? Concepts, methods and barriers to social impact assessment in Italian foundations," International Review on Public and Nonprofit Marketing, Springer;International Association of Public and Non-Profit Marketing, vol. 15(4), pages 553-574, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tim Benijts, 2014. "A Business Sustainability Model for Government Corporations. A Belgian Case Study," Business Strategy and the Environment, Wiley Blackwell, vol. 23(3), pages 204-216, March.
    2. Wasserman, Deborah L., 2010. "Using a systems orientation and foundational theory to enhance theory-driven human service program evaluations," Evaluation and Program Planning, Elsevier, vol. 33(2), pages 67-80, May.
    3. Peyton, David J. & Scicchitano, Michael, 2017. "Devil is in the details: Using logic models to investigate program process," Evaluation and Program Planning, Elsevier, vol. 65(C), pages 156-162.
    4. Matt Andrews, 2022. "This is How to Think About and Achieve Public Policy Success," CID Working Papers 413, Center for International Development at Harvard University.
    5. Wifo, 2021. "WIFO-Monatsberichte, Heft 10/2021," WIFO Monatsberichte (monthly reports), WIFO, vol. 94(10), October.
    6. Sobelson, Robyn K. & Young, Andrea C., 2013. "Evaluation of a federally funded workforce development program: The Centers for Public Health Preparedness," Evaluation and Program Planning, Elsevier, vol. 37(C), pages 50-57.
    7. Wu, Huang & Shen, Jianping & Jones, Jeffrey & Gao, Xingyuan & Zheng, Yunzheng & Krenn, Huilan Y., 2019. "Using logic model and visualization to conduct portfolio evaluation," Evaluation and Program Planning, Elsevier, vol. 74(C), pages 69-75.
    8. Vinícius P. Rodrigues & Daniela C. A. Pigosso & Jakob W. Andersen & Tim C. McAloone, 2018. "Evaluating the Potential Business Benefits of Ecodesign Implementation: A Logic Model Approach," Sustainability, MDPI, vol. 10(6), pages 1-26, June.
    9. Janger, Jürgen & Schubert, Torben & Andries, Petra & Rammer, Christian & Hoskens, Machteld, 2017. "The EU 2020 innovation indicator: A step forward in measuring innovation outputs and outcomes?," Research Policy, Elsevier, vol. 46(1), pages 30-42.
    10. Jürgen Janger & Agnes Kügler, 2018. "Innovationseffizienz. Österreich im internationalen Vergleich," WIFO Studies, WIFO, number 61111.
    11. Fangyuan Chang & Andrea Eriksson & Britt Östlund, 2020. "Discrepancies between Expected and Actual Implementation: The Process Evaluation of PERS Integration in Nursing Homes," IJERPH, MDPI, vol. 17(12), pages 1-18, June.
    12. Louise R. Manfredi & Meriel Stokoe & Rebecca Kelly & Seyeon Lee, 2021. "Teaching Sustainable Responsibility through Informal Undergraduate Design Education," Sustainability, MDPI, vol. 13(15), pages 1-13, July.
    13. Laura Kreiling & Ahmed Bounfour, 2020. "A practice-based maturity model for holistic TTO performance management: development and initial use," The Journal of Technology Transfer, Springer, vol. 45(6), pages 1718-1747, December.
    14. Hyun-Kyu KANG, 2015. "Development of Guideline for Preliminary Feasibility Study on Government R&D Programs in Korea," Proceedings of International Academic Conferences 2805212, International Institute of Social and Economic Sciences.
    15. Carlos Vallejo & David Romero & Arturo Molina, 2017. "Implementation of best manufacturing practices using logic models and system dynamics: project design and project assessment views," Information Systems and e-Business Management, Springer, vol. 15(2), pages 535-575, May.
    16. Kaplan, Sue A. & Garrett, Katherine E., 2005. "The use of logic models by community-based initiatives," Evaluation and Program Planning, Elsevier, vol. 28(2), pages 167-172, May.
    17. Lena Ries & Markus Beckmann & Peter Wehnert, 2023. "Sustainable smart product-service systems: a causal logic framework for impact design," Journal of Business Economics, Springer, vol. 93(4), pages 667-706, May.
    18. Johanna D. Birckmayer & Carol Hirschon Weiss, 2000. "Theory-Based Evaluation in Practice," Evaluation Review, , vol. 24(4), pages 407-431, August.
    19. Jürgen Janger & Tim Slickers, 2023. "Wissensproduktion und Wissensverwertung in Österreich im internationalen Vergleich," WIFO Monatsberichte (monthly reports), WIFO, vol. 96(10), pages 699-714, October.
    20. Levine, Roger & Russ-Eft, Darlene & Burling, Andrea & Stephens, Jennifer & Downey, Joanna, 2013. "Evaluating health services research capacity building programs: Implications for health services and human resource development," Evaluation and Program Planning, Elsevier, vol. 37(C), pages 1-11.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:36:y:2012:i:5:p:375-401. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.