IDEAS home Printed from https://ideas.repec.org/a/iab/iabjlr/v46i3p239-252.html
   My bibliography  Save this article

Replication issues in social experiments: lessons from US labor market programs

Author

Listed:
  • Barnow, Burt S.

    (Trachtenberg School of Public Policy and Public Administration, George Washington University, USA)

  • Greenberg, David

    (Department of Economics, University of Maryland Baltimore County, USA)

Abstract

"When evaluating a pilot or demonstration program, there are risks from drawing inferences from a single test. This paper reviews the experiences of replication efforts from demonstrations using randomized controlled trials in the initial evaluation and the replications. Although replications of promising programs are primarily gathered to increase sample size, replications are also used to learn if the intervention is successful for other target groups and geographic locations, and to vary some of the intervention's features. In many cases, replications fail to achieve the same success as the original evaluation, and the paper reviews reasons that have been suggested for such failures. The paper reviews what has been learned from replications where random assignment was used in six instances: income maintenance experiments, unemployment insurance bonus experiments, the Center for Employment Training program, job clubs, job search experiments, and the Quantum Opportunity Program. The paper concludes by summarizing lessons learned from the review and areas where more research is needed." (Author's abstract, IAB-Doku) ((en))

Suggested Citation

  • Barnow, Burt S. & Greenberg, David, 2013. "Replication issues in social experiments: lessons from US labor market programs," Journal for Labour Market Research, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany], vol. 46(3), pages 239-252.
    • Handle: RePEc:iab:iabjlr:v:46:i:3:p:239-252
    DOI: 10.1007/s12651-013-0133-2
    as

    Download full text from publisher

    File URL: https://doi.org/10.1007/s12651-013-0133-2
    Download Restriction: no
    File URL: https://libkey.io/10.1007/s12651-013-0133-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:mpr:mprres:1539 is not listed on IDEAS
    2. Kremer, Michael Robert & Miguel, Edward A., 2004. "The Illusion of Sustainability," Department of Economics, Working Paper Series qt94p8w1d7, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    3. Ashenfelter, Orley & Ashmore, David & Deschenes, Olivier, 2005. "Do unemployment insurance recipients actively seek work? Evidence from randomized trials in four U.S. States," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 53-75.
    4. Bruce D. Meyer, 1995. "Lessons from the U.S. Unemployment Insurance Experiments," Journal of Economic Literature, American Economic Association, vol. 33(1), pages 91-131, March.
    5. Allen Schirm & Nuria Rodriguez-Planas & Myles Maxfield & Christina Clark Tuttle, 2003. "The Quantum Opportunity Program Demonstration: Short-Term Impacts," Mathematica Policy Research Reports 3bb786088fbe413b8ea3fd69a, Mathematica Policy Research.
    6. Philip K. Robins & Robert G. Spiegelman (ed.), 2001. "Reemployment Bonuses in the Unemployment Insurance System: Evidence from Three Field Experiments," Books from Upjohn Press, W.E. Upjohn Institute for Employment Research, number rbuis.
    7. Michael Kremer & Edward Miguel, 2007. "The Illusion of Sustainability," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1007-1065.
    8. Myles Maxfield & Allen Schirm & Nuria Rodriguez-Planas, 2003. "The Quantum Opportunity Program Demonstration: Implementation and Short-Term Impacts," Mathematica Policy Research Reports 4ca5797760784df6a647d1131, Mathematica Policy Research.
    9. Woodbury, Stephen A & Spiegelman, Robert G, 1987. "Bonuses to Workers and Employers to Reduce Unemployment: Randomized Trials in Illinois," American Economic Review, American Economic Association, vol. 77(4), pages 513-530, September.
    10. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    11. Walter Corson & David Long & Walter Nicholson, 1985. "Evaluation of the Charleston Claimant Placement and Work Test Demonstration," Mathematica Policy Research Reports 9af520d14c1b4654b8080d304, Mathematica Policy Research.
    12. Walter Corson & Robert G. Spiegelman, "undated". "Design of Three Field Experiments," Mathematica Policy Research Reports 8449f3e499d54cee90a37ab29, Mathematica Policy Research.
    13. repec:mpr:mprres:3655 is not listed on IDEAS
    14. repec:mpr:mprres:3525 is not listed on IDEAS
    15. Gustavo J. Bobonis & Edward Miguel & Charu Puri-Sharma, 2006. "Anemia and School Participation," Journal of Human Resources, University of Wisconsin Press, vol. 41(4).
    16. repec:mpr:mprres:7757 is not listed on IDEAS
    17. Gary Burtless, 1986. "The work response to a guaranteed income: a survey of experimental evidence," Conference Series ; [Proceedings], Federal Reserve Bank of Boston, vol. 30, pages 22-59.
    18. Stephen A. Wandner, 2010. "Solving the Reemployment Puzzle: From Research to Policy," Books from Upjohn Press, W.E. Upjohn Institute for Employment Research, number srep.
    19. Bruce D. Meyer, 1992. "Policy Lessons from the U.S. Unemployment Experiments," NBER Working Papers 4197, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. David Greenberg & Burt S. Barnow, 2014. "Flaws in Evaluations of Social Programs," Evaluation Review, , vol. 38(5), pages 359-387, October.
    2. Larry L. Orr, 2015. "2014 Rossi Award Lecture," Evaluation Review, , vol. 39(2), pages 167-178, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Adhvaryu, Achyuta & Nyshadham, Anant, 2011. "Healthcare Choices, Information and Health Outcomes," Center Discussion Papers 107257, Yale University, Economic Growth Center.
    2. Meredith, Jennifer & Robinson, Jonathan & Walker, Sarah & Wydick, Bruce, 2013. "Keeping the doctor away: Experimental evidence on investment in preventative health products," Journal of Development Economics, Elsevier, vol. 105(C), pages 196-210.
    3. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    4. Christopher J. O'Leary, 2017. "Evaluating Public Employment Programs with Field Experiments: A Survey of American Evidence," Upjohn Working Papers 17-279, W.E. Upjohn Institute for Employment Research.
    5. William Easterly, 2009. "Can the West Save Africa?," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 373-447, June.
    6. Hoffmann, Bridget, 2018. "Do non-monetary prices target the poor? Evidence from a field experiment in India," Journal of Development Economics, Elsevier, vol. 133(C), pages 15-32.
    7. Fitzsimons, Emla & Malde, Bansi & Mesnard, Alice & Vera-Hernández, Marcos, 2016. "Nutrition, information and household behavior: Experimental evidence from Malawi," Journal of Development Economics, Elsevier, vol. 122(C), pages 113-126.
    8. Okeke, Edward N. & Adepiti, Clement A. & Ajenifuja, Kayode O., 2013. "What is the price of prevention? New evidence from a field experiment," Journal of Health Economics, Elsevier, vol. 32(1), pages 207-218.
    9. Fischer, Greg & Karlan, Dean & McConnell, Margaret & Raffler, Pia, 2019. "Short-term subsidies and seller type: A health products experiment in Uganda," Journal of Development Economics, Elsevier, vol. 137(C), pages 110-124.
    10. Baylis, Kathy & Ham, Andres, 2015. "How important is spatial correlation in randomized controlled trials?," 2015 AAEA & WAEA Joint Annual Meeting, July 26-28, San Francisco, California 205586, Agricultural and Applied Economics Association.
    11. Bonan, Jacopo & Battiston, Pietro & Bleck, Jaimie & LeMay-Boucher, Philippe & Pareglio, Stefano & Sarr, Bassirou & Tavoni, Massimo, 2021. "Social interaction and technology adoption: Experimental evidence from improved cookstoves in Mali," World Development, Elsevier, vol. 144(C).
    12. Antonia Grohmann & Sahra Sakha, 2015. "The Effect of Peer Observation on Consumption Choices: Experimental Evidence," Discussion Papers of DIW Berlin 1525, DIW Berlin, German Institute for Economic Research.
    13. Mobarak, Ahmed Musfiq & Rosenzweig, Mark, 2013. "Risk, Insurance and Wages in General Equilibrium," Working Papers 127, Yale University, Department of Economics.
    14. M. Daniele Paserman, 2008. "Job Search and Hyperbolic Discounting: Structural Estimation and Policy Evaluation," Economic Journal, Royal Economic Society, vol. 118(531), pages 1418-1452, August.
    15. Sarah Baird & Joan Hamory Hicks & Michael Kremer & Edward Miguel, 2016. "Worms at Work: Long-run Impacts of a Child Health Investment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(4), pages 1637-1680.
    16. Temple, Jonathan R.W., 2010. "Aid and Conditionality," Handbook of Development Economics, in: Dani Rodrik & Mark Rosenzweig (ed.), Handbook of Development Economics, edition 1, volume 5, chapter 0, pages 4415-4523, Elsevier.
    17. Pieter Gautier & Paul Muller & Bas van der Klaauw & Michael Rosholm & Michael Svarer, 2018. "Estimating Equilibrium Effects of Job Search Assistance," Journal of Labor Economics, University of Chicago Press, vol. 36(4), pages 1073-1125.
    18. Daniel Bennett, 2012. "Does Clean Water Make You Dirty?: Water Supply and Sanitation in the Philippines," Journal of Human Resources, University of Wisconsin Press, vol. 47(1), pages 146-173.
    19. Singh, Prakarsh, 2011. "Spillovers in learning and behavior: Evidence from a nutritional information campaign in urban slums," MPRA Paper 33362, University Library of Munich, Germany.
    20. Bijwaard, G.E., 2002. "Instrumental variable estimation for duration data," Econometric Institute Research Papers EI 2002-39, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iab:iabjlr:v:46:i:3:p:239-252. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: IAB, Geschäftsbereich Wissenschaftliche Fachinformation und Bibliothek (email available below). General contact details of provider: https://edirc.repec.org/data/iabbbde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.