IDEAS home Printed from https://ideas.repec.org/a/sae/anname/v587y2003i1p31-48.html
   My bibliography  Save this article

When can we Conclude that Treatments or Programs “Don’t Work†?

Author

Listed:
  • David Weisburd

    (Department of Criminology and Criminal Justice at the University of Maryland; Institute of Criminology at the Hebrew University Law School in Jerusalem)

  • Cynthia M. Lum
  • Sue-Ming Yang

Abstract

In this article, the authors examine common practices of reporting statistically nonsignificant findings in criminal justice evaluation studies. They find that criminal justice evaluators often make formal errors in the reporting of statistically nonsignificant results. Instead of simply concluding that the results were not statistically significant, or that there is not enough evidence to support an effect of treatment, they often mistakenly accept the null hypothesis and state that the intervention had no impact or did not work. The authors propose that researchers define a second null hypothesis that sets a minimal threshold for program effectiveness. In an illustration of this approach, they find that more than half of the studies that had no statistically significant finding for a traditional, no difference null hypothesis evidenced a statistically significant result in the case of a minimal worthwhile treatment effect null hypothesis.

Suggested Citation

  • David Weisburd & Cynthia M. Lum & Sue-Ming Yang, 2003. "When can we Conclude that Treatments or Programs “Don’t Work†?," The ANNALS of the American Academy of Political and Social Science, , vol. 587(1), pages 31-48, May.
    • Handle: RePEc:sae:anname:v:587:y:2003:i:1:p:31-48
    DOI: 10.1177/0002716202250782
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0002716202250782
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0002716202250782?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. David P. Farrington, 2003. "Methodological Quality Standards for Evaluation Research," The ANNALS of the American Academy of Political and Social Science, , vol. 587(1), pages 49-68, May.
    2. David Weisburd & Cynthia M. Lum & Anthony Petrosino, 2001. "Does Research Design Affect Study Outcomes in Criminal Justice?," The ANNALS of the American Academy of Political and Social Science, , vol. 578(1), pages 50-70, November.
    3. David Greenberg & Robert H. Meyer & Michael Wiseman, 1994. "Multisite Employment and Training Program Evaluations: A Tale of Three Studies," ILR Review, Cornell University, ILR School, vol. 47(4), pages 679-691, July.
    4. Levitt, Steven D, 1997. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime," American Economic Review, American Economic Association, vol. 87(3), pages 270-290, June.
    5. Sandra M. Nutley & Huw T.O. Davies, 1999. "The Fall and Rise of Evidence in Criminal Justice," Public Money & Management, Taylor & Francis Journals, vol. 19(1), pages 47-54, January.
    6. Joan McCord, 2003. "Cures That Harm: Unanticipated Outcomes of Crime Prevention Programs," The ANNALS of the American Academy of Political and Social Science, , vol. 587(1), pages 16-30, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Friedrich Lösel & Andreas Beelmann, 2003. "Effects of Child Skills Training in Preventing Antisocial Behavior: A Systematic Review of Randomized Evaluations," The ANNALS of the American Academy of Political and Social Science, , vol. 587(1), pages 84-109, May.
    2. Anthony Petrosino, 2003. "Standards for Evidence and Evidence for Standards: The Case of School-Based Drug Prevention," The ANNALS of the American Academy of Political and Social Science, , vol. 587(1), pages 180-207, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kevin Petersen & Robert C. Davis & David Weisburd & Bruce Taylor, 2022. "Effects of second responder programs on repeat incidents of family abuse: An updated systematic review and meta‐analysis," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(1), March.
    2. Friedrich Lösel & Andreas Beelmann, 2003. "Effects of Child Skills Training in Preventing Antisocial Behavior: A Systematic Review of Randomized Evaluations," The ANNALS of the American Academy of Political and Social Science, , vol. 587(1), pages 84-109, May.
    3. David Weisburd & John E. Eck, 2004. "What Can Police Do to Reduce Crime, Disorder, and Fear?," The ANNALS of the American Academy of Political and Social Science, , vol. 593(1), pages 42-65, May.
    4. Raphael, Steven & Winter-Ember, Rudolf, 2001. "Identifying the Effect of Unemployment on Crime," Journal of Law and Economics, University of Chicago Press, vol. 44(1), pages 259-283, April.
    5. Entorf, Horst & Spengler, Hannes, 2000. "Socioeconomic and demographic factors of crime in Germany: Evidence from panel data of the German states," International Review of Law and Economics, Elsevier, vol. 20(1), pages 75-106, March.
    6. Menusch Khadjavi, 2018. "Deterrence works for criminals," European Journal of Law and Economics, Springer, vol. 46(1), pages 165-178, August.
    7. Francesco Drago & Roberto Galbiati & Francesco Sobbrio, 2020. "The Political Cost of Being Soft on Crime: Evidence from a Natural Experiment," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 3305-3336.
    8. Liu, Xiaodong & Patacchini, Eleonora & Zenou, Yves & Lee, Lung-Fei, 2011. "Criminal Networks: Who is the Key Player?," Research Papers in Economics 2011:7, Stockholm University, Department of Economics.
    9. Blesse, Sebastian & Diegmann, André, 2022. "The place-based effects of police stations on crime: Evidence from station closures," Journal of Public Economics, Elsevier, vol. 207(C).
    10. Edward L. Glaeser, 2021. "Urban Resilience," NBER Working Papers 29261, National Bureau of Economic Research, Inc.
    11. Jaewook Byeon & Iljoong Kim & Dongwon Lee, 2018. "Protest and property crime: political use of police resources and the deterrence of crime," Public Choice, Springer, vol. 175(1), pages 181-196, April.
    12. Yahagi, Ken, 2021. "Law enforcement with motivated agents," International Review of Law and Economics, Elsevier, vol. 66(C).
    13. Almeida, Rita & Carneiro, Pedro, 2006. "Enforcement of Regulation, Informal Labour, Firm Size and Firm Performance," CEPR Discussion Papers 5976, C.E.P.R. Discussion Papers.
    14. Almeida, Rita K. & Carneiro, Pedro, 2005. "Enforcement of Regulation, Informal Labor and Firm Performance," IZA Discussion Papers 1759, Institute of Labor Economics (IZA).
    15. Vollaard, Ben & Koning, Pierre, 2009. "The effect of police on crime, disorder and victim precaution. Evidence from a Dutch victimization survey," International Review of Law and Economics, Elsevier, vol. 29(4), pages 336-348, December.
    16. Andrew Haughwout & Robert Inman & Steven Craig & Thomas Luce, 2004. "Local Revenue Hills: Evidence from Four U.S. Cities," The Review of Economics and Statistics, MIT Press, vol. 86(2), pages 570-585, May.
    17. Altindag, Duha T., 2012. "Crime and unemployment: Evidence from Europe," International Review of Law and Economics, Elsevier, vol. 32(1), pages 145-157.
    18. Isabel-María García-Sánchez & Luis Rodríguez-Domínguez & Javier Parra-Domínguez, 2013. "Yearly evolution of police efficiency in Spain and explanatory factors," Central European Journal of Operations Research, Springer;Slovak Society for Operations Research;Hungarian Operational Research Society;Czech Society for Operations Research;Österr. Gesellschaft für Operations Research (ÖGOR);Slovenian Society Informatika - Section for Operational Research;Croatian Operational Research Society, vol. 21(1), pages 31-62, January.
    19. Anthony Petrosino, 2003. "Estimates of Randomized Controlled Trials Across Six Areas of Childhood Intervention: A Bibliometric Analysis," The ANNALS of the American Academy of Political and Social Science, , vol. 589(1), pages 190-202, September.
    20. Stephen Machin & Olivier Marie, 2011. "Crime And Police Resources: The Street Crime Initiative," Journal of the European Economic Association, European Economic Association, vol. 9(4), pages 678-701, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:anname:v:587:y:2003:i:1:p:31-48. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.