IDEAS home Printed from https://ideas.repec.org/a/ejw/journl/v10y2013i1p97-107.html
   My bibliography  Save this article

We Agree That Statistical Significance Proves Essentially Nothing: A Rejoinder to Thomas Mayer

Author

Listed:
  • Stephen T. Ziliak
  • Deirdre N. McCloskey

Abstract

In several dozen journal reviews and in many other comments we have received—from, for example, four Nobel laureates, the statistician Dennis Lindley (2012), the statistician Arnold Zellner (2004), the mathematician Olle Häggström (2010), the sociologist Steve Fuller (2008), and the historian Theodore Porter (2008)—no one has ever tried to defend null hypothesis significance testing and its numerous errors. Recent articles by Thomas Mayer (2012, 2013), commenting on our book The Cult of Statistical Significance, are no exception. Of the five major claims we make in our book about the theory and practice of significance testing in economics, Mayer strongly agrees with four. On the fifth claim our disagreement is a matter of degree, not of kind, with no substantive change in results. Overall, Mayer agrees with us and with the new and growing consensus that statistical significance proves essentially nothing and has to change.

Suggested Citation

  • Stephen T. Ziliak & Deirdre N. McCloskey, 2013. "We Agree That Statistical Significance Proves Essentially Nothing: A Rejoinder to Thomas Mayer," Econ Journal Watch, Econ Journal Watch, vol. 10(1), pages 97-107, January.
    • Handle: RePEc:ejw:journl:v:10:y:2013:i:1:p:97-107
    as

    Download full text from publisher

    File URL: https://econjwatch.org/File+download/623/ZiliakMcCloskeyJan2013.pdf?mimetype=pdf
    Download Restriction: no
    File URL: https://econjwatch.org/849
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    2. Stephen T. Ziliak & Deirdre N. McCloskey, 2004. "Size Matters: The Standard Error of Regressions in the American Economic Review," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 331-358, August.
    3. Elliott, Graham & Granger, Clive W.J., 2004. "Evaluating significance: comments on "size matters"," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 547-550, November.
    4. Kevin Hoover & Mark Siegler, 2008. "Sound and fury: McCloskey and significance testing in economics," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 1-37.
    5. Zellner, Arnold, 2004. "To test or not to test and if so, how?: Comments on "size matters"," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 581-586, November.
    6. McCloskey, Donald N, 1985. "The Loss Function Has Been Mislaid: The Rhetoric of Significance Tests," American Economic Review, American Economic Association, vol. 75(2), pages 201-205, May.
    7. Walter Krämer, 2011. "The Cult of Statistical Significance – What Economists Should and Should Not Do to Make their Data Talk," Schmollers Jahrbuch : Journal of Applied Social Science Studies / Zeitschrift für Wirtschafts- und Sozialwissenschaften, Duncker & Humblot, Berlin, vol. 131(3), pages 455-468.
    8. Thomas Mayer, 2013. "Reply to Deirdre McCloskey and Stephen Ziliak on Statistical Significance," Econ Journal Watch, Econ Journal Watch, vol. 10(1), pages 87-96, January.
    9. Deirdre McCloskey & Stephen Ziliak, 2008. "Signifying nothing: reply to Hoover and Siegler," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 39-55.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wang, Yang & Ashton, John K. & Jaafar, Aziz, 2019. "Money shouts! How effective are punishments for accounting fraud?," The British Accounting Review, Elsevier, vol. 51(5).
    2. Nektarios A. Michail & Constantinos I. Massouras, 2014. "Back to Basics: Is Statistical Significance all that Matters?," Working Papers 2014-3, Central Bank of Cyprus.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thomas Mayer, 2012. "Ziliak and McCloskey's Criticisms of Significance Tests: An Assessment," Econ Journal Watch, Econ Journal Watch, vol. 9(3), pages 256-297, September.
    2. Peter J. Veazie, 2015. "Understanding Statistical Testing," SAGE Open, , vol. 5(1), pages 21582440145, January.
    3. Kevin Hoover & Mark Siegler, 2008. "Sound and fury: McCloskey and significance testing in economics," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 1-37.
    4. Nektarios A. Michail & Constantinos I. Massouras, 2014. "Back to Basics: Is Statistical Significance all that Matters?," Working Papers 2014-3, Central Bank of Cyprus.
    5. John D. Levendis, 2018. "Time Series Econometrics," Springer Texts in Business and Economics, Springer, number 978-3-319-98282-3, April.
    6. Tom Engsted, 2009. "Statistical vs. Economic Significance in Economics and Econometrics: Further comments on McCloskey & Ziliak," CREATES Research Papers 2009-17, Department of Economics and Business Economics, Aarhus University.
    7. Black, Bernard & Hollingsworth, Alex & Nunes, Letícia & Simon, Kosali, 2022. "Simulated power analyses for observational studies: An application to the Affordable Care Act Medicaid expansion," Journal of Public Economics, Elsevier, vol. 213(C).
    8. Timothy R. Wojan & Jason P. Brown & Dayton M. Lambert, 2014. "What to Do about the "Cult of Statistical Significance"? A Renewable Fuel Application using the Neyman-Pearson Protocol," Applied Economic Perspectives and Policy, Agricultural and Applied Economics Association, vol. 36(4), pages 674-695.
    9. Jens Rommel & Meike Weltin, 2021. "Is There a Cult of Statistical Significance in Agricultural Economics?," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(3), pages 1176-1191, September.
    10. Altman, Morris, 2004. "Statistical significance, path dependency, and the culture of journal publication," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 651-663, November.
    11. Thomas Mayer, 2012. "Ziliak and McClosky?s Criticisms of Significance Tests: A Damage Assessment," Working Papers 61, University of California, Davis, Department of Economics.
    12. Mbatha, C.N. & Gustafsson, M.A., 2013. "The standard error of regressions: a note on new evidence of significance misuse," Agrekon, Agricultural Economics Association of South Africa (AEASA), vol. 52(1), March.
    13. repec:ags:ijag24:345236 is not listed on IDEAS
    14. Altman, Morris, 2020. "A more scientific approach to applied economics: Reconstructing statistical, analytical significance, and correlation analysis," Economic Analysis and Policy, Elsevier, vol. 66(C), pages 315-324.
    15. Kim, Jae H. & Ji, Philip Inyeob, 2015. "Significance testing in empirical finance: A critical review and assessment," Journal of Empirical Finance, Elsevier, vol. 34(C), pages 1-14.
    16. Thomas Mayer, 2012. "Ziliak and McClosky?s Criticisms of Significance Tests: A Damage Assessment," Working Papers 126, University of California, Davis, Department of Economics.
    17. Nicolas Vallois & Dorian Jullien, 2017. "Estimating Rationality in Economics: A History of Statistical Methods in Experimental Economics," GREDEG Working Papers 2017-20, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    18. Jason P Brown & Dayton M Lambert & Timothy R Wojan, 2019. "The Effect of the Conservation Reserve Program on Rural Economies: Deriving a Statistical Verdict from a Null Finding," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 101(2), pages 528-540.
    19. Soyer, Emre & Hogarth, Robin M., 2012. "The illusion of predictability: How regression statistics mislead experts," International Journal of Forecasting, Elsevier, vol. 28(3), pages 695-711.
    20. Meszaros, Sandor, 2008. "Theory testing (hypothesis testing) in agricultural economics," Studies in Agricultural Economics, Research Institute for Agricultural Economics, vol. 107, pages 1-13, March.
    21. Niclas Berggren & Mikael Elinder & Henrik Jordahl, 2008. "Trust and growth: a shaky relationship," Empirical Economics, Springer, vol. 35(2), pages 251-274, September.

    More about this item

    Keywords

    Significance test; economic significance; Student’s t; oomph; Gosset; Fisher;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ejw:journl:v:10:y:2013:i:1:p:97-107. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jason Briggeman (email available below). General contact details of provider: https://edirc.repec.org/data/edgmuus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.