IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v39y2015i2p167-178.html
   My bibliography  Save this article

2014 Rossi Award Lecture

Author

Listed:
  • Larry L. Orr

Abstract

Background: For much of the last 40 years, the evaluation profession has been consumed in a battle over internal validity. Today, that battle has been decided. Random assignment, while still far from universal in practice, is almost universally acknowledged as the preferred method for impact evaluation. It is time for the profession to shift its attention to the remaining major flaws in the “standard model†of evaluation: (i) external validity and (ii) the high cost and low hit rate of experimental evaluations as currently practiced. Recommendations: To raise the profession’s attention to external validity, the author recommends some simple, easy steps to be taken in every evaluation. The author makes two recommendations to increase the number of interventions found to be effective within existing resources: First, a two-stage evaluation strategy in which a cheap, streamlined Stage 1 evaluation is followed by a more intensive Stage 2 evaluation only for those interventions found to be effective in a Stage 1 trial and, second, use of random assignment to guide the myriad program management decisions that must be made in the course of routine program operations. This article is not intended as a solution to these issues: It is intended to stimulate the evaluation community to take these issues more seriously and to develop innovative solutions.

Suggested Citation

  • Larry L. Orr, 2015. "2014 Rossi Award Lecture," Evaluation Review, , vol. 39(2), pages 167-178, April.
    • Handle: RePEc:sae:evarev:v:39:y:2015:i:2:p:167-178
    DOI: 10.1177/0193841X15573659
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X15573659
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X15573659?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert B. Olsen & Larry L. Orr & Stephen H. Bell & Elizabeth A. Stuart, 2013. "External Validity in Policy Evaluations That Choose Sites Purposively," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 32(1), pages 107-121, January.
    2. repec:mpr:mprres:8169 is not listed on IDEAS
    3. Douglas J. Besharov, 2009. "Presidential address: From the Great Society to continuous improvement government: Shifting from “does it work?” to “what would make it better?”," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 28(2), pages 199-220.
    4. repec:mpr:mprres:8164 is not listed on IDEAS
    5. repec:mpr:mprres:3655 is not listed on IDEAS
    6. repec:mpr:mprres:6574 is not listed on IDEAS
    7. repec:mpr:mprres:6097 is not listed on IDEAS
    8. repec:mpr:mprres:6676 is not listed on IDEAS
    9. Bruce D. Meyer, 1995. "Lessons from the U.S. Unemployment Insurance Experiments," Journal of Economic Literature, American Economic Association, vol. 33(1), pages 91-131, March.
    10. Allen Schirm & Nuria Rodriguez-Planas & Myles Maxfield & Christina Clark Tuttle, 2003. "The Quantum Opportunity Program Demonstration: Short-Term Impacts," Mathematica Policy Research Reports 3bb786088fbe413b8ea3fd69a, Mathematica Policy Research.
    11. Burt S. Barnow & David Greenberg, 2013. "Replication issues in social experiments: lessons from US labor market programs [Probleme bei sozialen Experimenten: Lehren aus US-amerikanischen Arbeitsmarktprogrammen]," Journal for Labour Market Research, Springer;Institute for Employment Research/ Institut für Arbeitsmarkt- und Berufsforschung (IAB), vol. 46(3), pages 239-252, September.
    12. Myles Maxfield & Allen Schirm & Nuria Rodriguez-Planas, 2003. "The Quantum Opportunity Program Demonstration: Implementation and Short-Term Impacts," Mathematica Policy Research Reports 4ca5797760784df6a647d1131, Mathematica Policy Research.
    13. repec:mpr:mprres:7293 is not listed on IDEAS
    14. Michael J. Puma & Nancy R. Burstein, 1994. "The national evaluation of the food stamp employment and training program," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 13(2), pages 311-330.
    15. repec:mpr:mprres:3525 is not listed on IDEAS
    16. repec:mpr:mprres:6720 is not listed on IDEAS
    17. Peter Z. Schochet & John Burghardt & Sheena McConnell, 2008. "Does Job Corps Work? Impact Findings from the National Job Corps Study," American Economic Review, American Economic Association, vol. 98(5), pages 1864-1886, December.
    18. Daniel Gubits & Rachel Cook & Stephen Bell & Michelle Derr & Jillian Berk & Ann Person & David Stapleton & Denise Hoffman & David Wittenburg, "undated". "BOND Implementation and Evaluation: Stage 2 Early Assessment Report," Mathematica Policy Research Reports bad563e55a8847b6b556ff64d, Mathematica Policy Research.
    19. Allen Schirm & Elizabeth Stuart & Allison McKie, "undated". "The Quantum Opportunity Program Demonstration: Final Impacts," Mathematica Policy Research Reports ac481c0313ec4f7bbf036f563, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rodríguez-Planas, Núria, 2017. "School, drugs, mentoring, and peers: Evidence from a randomized trial in the US," Journal of Economic Behavior & Organization, Elsevier, vol. 139(C), pages 166-181.
    2. Barnow, Burt S. & Greenberg, David, 2013. "Replication issues in social experiments: lessons from US labor market programs," Journal for Labour Market Research, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany], vol. 46(3), pages 239-252.
    3. Matthew Morton & Paul Montgomery, 2011. "Youth Empowerment Programs for Improving Self‐Efficacy and Self‐Esteem of Adolescents," Campbell Systematic Reviews, John Wiley & Sons, vol. 7(1), pages 1-80.
    4. Rodríguez-Planas, Núria, 2010. "Longer-Term Impacts of Mentoring, Educational Services, and Incentives to Learn: Evidence from a Randomized Trial," IZA Discussion Papers 4754, Institute of Labor Economics (IZA).
    5. Marios Michaelides & Peter Mueser & Jeffrey Smith, 2019. "Youth Unemployment and U.S. Job Search Assistance Policy during the Great Recession," University of Cyprus Working Papers in Economics 13-2019, University of Cyprus Department of Economics.
    6. Bean, Corliss N. & Kendellen, Kelsey & Halsall, Tanya & Forneris, Tanya, 2015. "Putting program evaluation into practice: Enhancing the Girls Just Wanna Have Fun program," Evaluation and Program Planning, Elsevier, vol. 49(C), pages 31-40.
    7. Núria Rodríquez-Planas, 2010. "Mentoring, Educational Services, and Economic Incentives Longer-term Evidence on Risky Behaviors from a Randomized Trial," Working Papers 462, Barcelona School of Economics.
    8. Patrick Tolan & David Henry & Michael Schoeny & Arin Bass & Peter Lovegrove & Emily Nichols, 2013. "Mentoring Interventions to Affect Juvenile Delinquency and Associated Problems: A Systematic Review," Campbell Systematic Reviews, John Wiley & Sons, vol. 9(1), pages 1-158.
    9. Rothstein, Jesse & Von Wachter, Till, 2016. "Social Experiments in the Labor Market," Department of Economics, Working Paper Series qt7957p9g6, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    10. Marios Michaelides & Peter R. Mueser & Jeffrey A. Smith, 2021. "Do Reemployment Programs For The Unemployed Work For Youth? Evidence From The Great Recession In The United States," Economic Inquiry, Western Economic Association International, vol. 59(1), pages 162-185, January.
    11. David Greenberg & Burt S. Barnow, 2014. "Flaws in Evaluations of Social Programs," Evaluation Review, , vol. 38(5), pages 359-387, October.
    12. Allen Schirm & Elizabeth Stuart & Allison McKie, "undated". "The Quantum Opportunity Program Demonstration: Final Impacts," Mathematica Policy Research Reports ac481c0313ec4f7bbf036f563, Mathematica Policy Research.
    13. Robert Kornfeld & Howard S. Bloom, 1997. "Measuring Program Impacts On Earnings and Employment," JCPR Working Papers 11, Northwestern University/University of Chicago Joint Center for Poverty Research.
    14. Boone, Jan & Sadrieh, Abdolkarim & van Ours, Jan C., 2009. "Experiments on unemployment benefit sanctions and job search behavior," European Economic Review, Elsevier, vol. 53(8), pages 937-951, November.
    15. Jeffrey Smith, 2000. "A Critical Survey of Empirical Methods for Evaluating Active Labor Market Policies," Swiss Journal of Economics and Statistics (SJES), Swiss Society of Economics and Statistics (SSES), vol. 136(III), pages 247-268, September.
    16. Marisa Shenk & Gina Livermore, "undated". "Work-Related Overpayments Among Disability Beneficiaries," Mathematica Policy Research Reports 04aef12ea834409c8be30dc5f, Mathematica Policy Research.
    17. Cockx, Bart & Robin, Stéphane R. & Goebel, Christian, 2006. "Income Support Policies for Part-Time Workers: A Stepping-Stone to Regular Jobs? An Application to Young Long-Term Unemployed Women in Belgium," IZA Discussion Papers 2432, Institute of Labor Economics (IZA).
    18. Nicola Pavoni & G. L. Violante, 2007. "Optimal Welfare-to-Work Programs," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 74(1), pages 283-318.
    19. María laura Alzúa & Guillermo Cruces & Carolina Lopez, 2016. "Long-Run Effects Of Youth Training Programs: Experimental Evidence From Argentina," Economic Inquiry, Western Economic Association International, vol. 54(4), pages 1839-1859, October.
    20. Burt S. Barnow & Jeffrey Smith, 2015. "Employment and Training Programs," NBER Chapters, in: Economics of Means-Tested Transfer Programs in the United States, Volume 2, pages 127-234, National Bureau of Economic Research, Inc.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:39:y:2015:i:2:p:167-178. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.