IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v47y2023i1p71-103.html
   My bibliography  Save this article

Using Multi-Arm Designs to Test Operating Welfare-to-Work Programs

Author

Listed:
  • Judith M. Gueron
  • Gayle Hamilton

Abstract

Background: In the early 1970s, most researchers thought that randomized controlled trials (RCTs) could not be used to measure the effectiveness of large-scale operating welfare reform and employment programs. By the mid-1970s, the Supported Work Demonstration showed that, under certain conditions, this was both feasible and valuable. However, the experimental design was simple; a multi-arm test had been rejected as unrealistic. Within 10 years, a three-arm design was implemented in San Diego to assess both a welfare-to-work program’s overall impact and the contribution of a specific component. Less than 10 years later, the Job Opportunities and Basic Skills Training (JOBS)/National Evaluation of Welfare-to-Work Strategies (NEWWS) study used a more complex design to determine the relative effectiveness of two strategies operated in the same locations: one emphasizing getting a job quickly and the other requiring basic education. In San Diego and JOBS/NEWWS, the tested reforms emerged from political processes and were funded through regular program budgets. In both cases, researchers inserted multi-arm RCTs into operating welfare offices, trading control over the treatment for scale (thousands of people) and real-world conditions. Both RCTs were successfully implemented. Objectives and Results: This article examines why multi-arm designs were attempted, how they were structured, why public administrators cooperated, what various actors sought to learn, and how the researchers determined what strategies the different experimental arms ended up to truly represent. The article concludes that these designs provide convincing evidence and can be inserted into operating programs if the studies address questions that are of keen and immediate interest to state or local program administrators and researchers.

Suggested Citation

  • Judith M. Gueron & Gayle Hamilton, 2023. "Using Multi-Arm Designs to Test Operating Welfare-to-Work Programs," Evaluation Review, , vol. 47(1), pages 71-103, February.
    • Handle: RePEc:sae:evarev:v:47:y:2023:i:1:p:71-103
    DOI: 10.1177/0193841X20984577
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X20984577
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X20984577?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. V. Joseph Hotz & Guido W. Imbens & Jacob A. Klerman, 2006. "Evaluating the Differential Effects of Alternative Welfare-to-Work Training Components: A Reanalysis of the California GAIN Program," Journal of Labor Economics, University of Chicago Press, vol. 24(3), pages 521-566, July.
    2. repec:mpr:mprres:6192 is not listed on IDEAS
    3. Michael Wiseman, 1991. "Research and policy: A symposium on the family support act of 1988," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 10(4), pages 588-589.
    4. Dehejia, Rajeev H, 2003. "Was There a Riverside Miracle? A Hierarchical Framework for Evaluating Programs with Grouped Data," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 1-11, January.
    5. Howard S. Bloom & Carolyn J. Hill & James A. Riccio, 2003. "Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 22(4), pages 551-575.
    6. Erica B. Baum, 1991. "When the witch doctors agree: The family support act and social science research," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 10(4), pages 603-615.
    7. Ron Haskins, 1991. "Congress writes a law: Research and welfare reform," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 10(4), pages 616-632.
    8. Michael Wiseman, 1991. "Research and policy: An afterword for the symposium on the family support act of 1988," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 10(4), pages 657-666.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Burt S. Barnow & Jeffrey Smith, 2015. "Employment and Training Programs," NBER Chapters, in: Economics of Means-Tested Transfer Programs in the United States, Volume 2, pages 127-234, National Bureau of Economic Research, Inc.
    2. Benjamin Lu & Eli Ben-Michael & Avi Feller & Luke Miratrix, 2023. "Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 420-453, August.
    3. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    4. Douglas J. Besharov, 2009. "Presidential address: From the Great Society to continuous improvement government: Shifting from “does it work?” to “what would make it better?”," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 28(2), pages 199-220.
    5. Richard Dorsett & Philip K. Robins, 2013. "A Multilevel Analysis of the Impacts of Services Provided by the U.K. Employment Retention and Advancement Demonstration," Evaluation Review, , vol. 37(2), pages 63-108, April.
    6. Hunt Allcott, 2012. "Site Selection Bias in Program Evaluation," NBER Working Papers 18373, National Bureau of Economic Research, Inc.
    7. DAVID H. Greenberg, 1992. "Conceptual Issues In Cost/Benefit Analysis Of Welfare‐To‐Work Programs," Contemporary Economic Policy, Western Economic Association International, vol. 10(4), pages 51-64, October.
    8. David Greenberg & Marvin Mandell & Matthew Onstott, 2000. "The dissemination and utilization of welfare-to-work experiments in state policymaking," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 19(3), pages 367-382.
    9. Straßheim, Holger, 2001. "Der Ruf der Sirenen - Zur Dynamik politischen Benchmarkings: Eine Analyse anhand der US-Sozialreformen," Discussion Papers, Research Unit: Regulation of Work FS II 01-201, WZB Berlin Social Science Center.
    10. Oscar Mitnik, 2008. "How do Training Programs Assign Participants to Training? Characterizing the Assignment Rules of Government Agencies for Welfare-to-Work Programs in California," Working Papers 0907, University of Miami, Department of Economics.
    11. Michael Wiseman, 1991. "Research and policy: An afterword for the symposium on the family support act of 1988," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 10(4), pages 657-666.
    12. Andrew P. Jaciw, 2016. "Assessing the Accuracy of Generalized Inferences From Comparison Group Studies Using a Within-Study Comparison Approach," Evaluation Review, , vol. 40(3), pages 199-240, June.
    13. Rajeev Dehejia, 2013. "The Porous Dialectic: Experimental and Non-Experimental Methods in Development Economics," WIDER Working Paper Series wp-2013-011, World Institute for Development Economic Research (UNU-WIDER).
    14. Susan Athey & Guido W. Imbens & Stefan Wager, 2018. "Approximate residual balancing: debiased inference of average treatment effects in high dimensions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(4), pages 597-623, September.
    15. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2008. "Nonparametric Tests for Treatment Effect Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 389-405, August.
    16. Dhaval Dave & Hope Corman & Nancy Reichman, 2012. "Effects of Welfare Reform on Education Acquisition of Adult Women," Journal of Labor Research, Springer, vol. 33(2), pages 251-282, June.
    17. Geoffrey L. Wallace & Robert Haveman, 2007. "The implications of differences between employer and worker employment|earnings reports for policy evaluation," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 26(4), pages 737-754.
    18. Chung Choe & Alfonso Flores-Lagunes & Sang-Jun Lee, 2015. "Do dropouts with longer training exposure benefit from training programs? Korean evidence employing methods for continuous treatments," Empirical Economics, Springer, vol. 48(2), pages 849-881, March.
    19. David Neumark & Brian Asquith & Brittany Bass, 2020. "Longer‐Run Effects Of Anti‐Poverty Policies On Disadvantaged Neighborhoods," Contemporary Economic Policy, Western Economic Association International, vol. 38(3), pages 409-434, July.
    20. Helen Lee & Sarah Shea Crowne & Melanie Estarziau & Keith Kranker & Charles Michalopoulos & Anne Warren & Tod Mijanovich & Jill H. Filene & Anne Duggan & Virginia Knox, "undated". "The Effects of Home Visiting on Prenatal Health, Birth Outcomes, and Health Care Use in the First Year of Life: Final Implementation and Impact Findings from the Mother and Infant Home Visiting Progra," Mathematica Policy Research Reports a9626a8d90bf4f01811d0c9d7, Mathematica Policy Research.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:47:y:2023:i:1:p:71-103. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.