IDEAS home Printed from https://ideas.repec.org/a/wly/jpamgt/v33y2014i3p778-808.html
   My bibliography  Save this article

A Conceptual Framework For Studying The Sources Of Variation In Program Effects

Author

Listed:
  • Michael J. Weiss
  • Howard S. Bloom
  • Thomas Brock

Abstract

Evaluations of public programs in many fields reveal that different types of programs—or different versions of the same program—vary in their effectiveness. Moreover, a program that is effective for one group of people might not be effective for other groups, and a program that is effective in one set of circumstances may not be effective in other circumstances. This paper presents a conceptual framework for research on such variation in program effects and the sources of this variation. The framework is intended to help researchers—both those who focus mainly on studying program implementation and those who focus mainly on estimating program effects—see how their respective pieces fit together in a way that helps to identify factors that explain variation in program effects, and thereby support more systematic data collection. The ultimate goal of the framework is to enable researchers to offer better guidance to policymakers and program operators on the conditions and practices that are associated with larger and more positive effects.

Suggested Citation

  • Michael J. Weiss & Howard S. Bloom & Thomas Brock, 2014. "A Conceptual Framework For Studying The Sources Of Variation In Program Effects," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(3), pages 778-808, June.
    • Handle: RePEc:wly:jpamgt:v:33:y:2014:i:3:p:778-808
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1002/pam.21760
    Download Restriction: no
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Omar Al-Ubaydli & John A. List & Dana Suskind, 2019. "The Science of Using Science: Towards an Understanding of the Threats to Scaling Experiments," NBER Working Papers 25848, National Bureau of Economic Research, Inc.
    2. Buhl-Wiggers, Julie & Kerwin, Jason T. & Muñoz-Morales, Juan & Smith, Jeffrey & Thornton, Rebecca, 2024. "Some children left behind: Variation in the effects of an educational intervention," Journal of Econometrics, Elsevier, vol. 243(1).
    3. Xu Qin & Guanglei Hong, 2017. "A Weighting Method for Assessing Between-Site Heterogeneity in Causal Mediation Mechanism," Journal of Educational and Behavioral Statistics, , vol. 42(3), pages 308-340, June.
    4. Sarah Dolfin & Nan Maxwell & Ankita Patnaik, "undated". "WHD Compliance Strategies: Directions for Future Research," Mathematica Policy Research Reports b7a5ca876e0b448f9b9c0850e, Mathematica Policy Research.
    5. Benjamin Lu & Eli Ben-Michael & Avi Feller & Luke Miratrix, 2023. "Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 420-453, August.
    6. Philip M. Gleason, "undated". "What's the Secret Ingredient? Searching for Policies and Practices that Make Charter Schools Successful," Mathematica Policy Research Reports eea6e24d9bf1409f92f60ae29, Mathematica Policy Research.
    7. Adam Gamoran, 2018. "Evidence-Based Policy in the Real World: A Cautionary View," The ANNALS of the American Academy of Political and Social Science, , vol. 678(1), pages 180-191, July.
    8. Trang Quynh Nguyen & Benjamin Ackerman & Ian Schmid & Stephen R Cole & Elizabeth A Stuart, 2018. "Sensitivity analyses for effect modifiers not observed in the target population when generalizing treatment effects from a randomized controlled trial: Assumptions, models, effect scales, data scenari," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-17, December.
    9. Jason Shumberger & Akheil Singla, 2022. "Are tax and expenditure limitations constraining institutions or institutionally irrelevant? Evidence from Minnesota," Public Budgeting & Finance, Wiley Blackwell, vol. 42(4), pages 3-33, December.
    10. Elizabeth Tipton & Robert B. Olsen, "undated". "Enhancing the Generalizability of Impact Studies in Education," Mathematica Policy Research Reports 35d5625333dc480aba9765b3b, Mathematica Policy Research.
    11. Robert Ammerman & Anne Duggan & John List & Lauren Supplee & Dana Suskind, 2021. "The role of open science practices in scaling evidence-based prevention programs," Natural Field Experiments 00741, The Field Experiments Website.
    12. Nianbo Dong & Benjamin M. Kelcey, 2020. "A Review of Causality in a Social World: Moderation, Mediation, and Spill-Over," Journal of Educational and Behavioral Statistics, , vol. 45(3), pages 374-378, June.
    13. Xu Qin & Jonah Deutsch & Guanglei Hong, 2021. "Unpacking Complex Mediation Mechanisms And Their Heterogeneity Between Sites In A Job Corps Evaluation," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 40(1), pages 158-190, January.
    14. Jeffrey Smith, 2022. "Treatment Effect Heterogeneity," Evaluation Review, , vol. 46(5), pages 652-677, October.
    15. Robert C. Granger, 2018. "The Roles Foundations Are Playing in the Evidence-Based Policy Movement," The ANNALS of the American Academy of Political and Social Science, , vol. 678(1), pages 145-154, July.
    16. Howard S. Bloom & Rebecca Unterman & Pei Zhu & Sean F. Reardon, 2020. "Lessons from New York City's Small Schools of Choice about High School Features that Promote Graduation for Disadvantaged Students," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(3), pages 740-771, June.
    17. Esterling, Kevin & Brady, David & Schwitzgebel, Eric, 2021. "The Necessity of Construct and External Validity for Generalized Causal Claims," OSF Preprints 2s8w5, Center for Open Science.
    18. Esterling, Kevin M. & Brady, David & Schwitzgebel, Eric, 2023. "The Necessity of Construct and External Validity for Generalized Causal Claims," I4R Discussion Paper Series 18, The Institute for Replication (I4R).
    19. Rebecca Maynard, 2024. "Improving the Usefulness and Use of Meta-Analysis to Inform Policy and Practice," Evaluation Review, , vol. 48(3), pages 515-543, June.
    20. Rafael Quintana, 2023. "Embracing complexity in social science research," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(1), pages 15-38, February.
    21. Christopher Rhoads, 2017. "Coherent Power Analysis in Multilevel Studies Using Parameters From Surveys," Journal of Educational and Behavioral Statistics, , vol. 42(2), pages 166-194, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:jpamgt:v:33:y:2014:i:3:p:778-808. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www3.interscience.wiley.com/journal/34787/home .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.