IDEAS home Printed from https://ideas.repec.org/a/sae/socres/v21y2016i2p136-147.html
   My bibliography  Save this article

Damaging the Case for Improving Social Science Methodology through Misrepresentation: Re-Asserting Confidence in Hypothesis Testing as a Valid Scientific Process

Author

Listed:
  • James Nicholson
  • Sean Mccusker

Abstract

This paper is a response to Gorard's article, ‘Damaging real lives through obstinacy: re-emphasising why significance testing is wrong’ in Sociological Research Online 21(1). For many years Gorard has criticised the way hypothesis tests are used in social science, but recently he has gone much further and argued that the logical basis for hypothesis testing is flawed: that hypothesis testing does not work, even when used properly. We have sympathy with the view that hypothesis testing is often carried out in social science contexts when it should not be, and that outcomes are often described in inappropriate terms, but this does not mean the theory of hypothesis testing, or its use, is flawed per se. There needs to be evidence to support such a contention. Gorard claims that: ‘Anyone knowing the problems, as described over one hundred years, who continues to teach, use or publish significance tests is acting unethically, and knowingly risking the damage that ensues.’ This is a very strong statement which impugns the integrity, not just the competence, of a large number of highly respected academics. We argue that the evidence he puts forward in this paper does not stand up to scrutiny: that the paper misrepresents what hypothesis tests claim to do, and uses a sample size which is far too small to discriminate properly a 10% difference in means in a simulation he constructs. He then claims that this simulates emotive contexts in which a 10% difference would be important to detect, implicitly misrepresenting the simulation as a reasonable model of those contexts.

Suggested Citation

  • James Nicholson & Sean Mccusker, 2016. "Damaging the Case for Improving Social Science Methodology through Misrepresentation: Re-Asserting Confidence in Hypothesis Testing as a Valid Scientific Process," Sociological Research Online, , vol. 21(2), pages 136-147, May.
  • Handle: RePEc:sae:socres:v:21:y:2016:i:2:p:136-147
    DOI: 10.5153/sro.3985
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.5153/sro.3985
    Download Restriction: no

    File URL: https://libkey.io/10.5153/sro.3985?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    2. Stephen Gorard, 2016. "Damaging Real Lives through Obstinacy: Re-Emphasising Why Significance Testing is Wrong," Sociological Research Online, , vol. 21(1), pages 102-115, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thees F Spreckelsen & Mariska Van Der Horst, 2016. "Is Banning Significance Testing the Best Way to Improve Applied Social Science Research? – Questions on Gorard (2016)," Sociological Research Online, , vol. 21(3), pages 95-105, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jyotirmoy Sarkar, 2018. "Will P†Value Triumph over Abuses and Attacks?," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 7(4), pages 66-71, July.
    2. Chatelain, Jean-Bernard & Ralf, Kirsten, 2018. "Publish and Perish: Creative Destruction and Macroeconomic Theory," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 46(2), pages 65-101.
    3. Segurado, Pedro & Gutiérrez-Cánovas, Cayetano & Ferreira, Teresa & Branco, Paulo, 2022. "Stressor gradient coverage affects interaction identification," Ecological Modelling, Elsevier, vol. 472(C).
    4. Uwe Hassler & Marc‐Oliver Pohle, 2022. "Unlucky Number 13? Manipulating Evidence Subject to Snooping," International Statistical Review, International Statistical Institute, vol. 90(2), pages 397-410, August.
    5. Kim, Jae H., 2017. "Stock returns and investors' mood: Good day sunshine or spurious correlation?," International Review of Financial Analysis, Elsevier, vol. 52(C), pages 94-103.
    6. Gergely Ganics & Atsushi Inoue & Barbara Rossi, 2021. "Confidence Intervals for Bias and Size Distortion in IV and Local Projections-IV Models," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 307-324, January.
    7. Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia & Jantsch, Antje, 2020. "Can p-values be meaningfully interpreted without random sampling?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 14, pages 71-91.
    8. Oliver Schilke & Sheen S. Levine & Olenka Kacperczyk & Lynne G. Zucker, 2019. "Call for Papers-Special Issue on Experiments in Organizational Theory," Organization Science, INFORMS, vol. 30(1), pages 232-234, February.
    9. Lopez, Belen & Rangel, Celia & Fernández, Manuel, 2022. "The impact of corporate social responsibility strategy on the management and governance axis for sustainable growth," Journal of Business Research, Elsevier, vol. 150(C), pages 690-698.
    10. Michaelides, Michael, 2021. "Large sample size bias in empirical finance," Finance Research Letters, Elsevier, vol. 41(C).
    11. Kelter, Riko, 2022. "Power analysis and type I and type II error rates of Bayesian nonparametric two-sample tests for location-shifts based on the Bayes factor under Cauchy priors," Computational Statistics & Data Analysis, Elsevier, vol. 165(C).
    12. Scott, E. Marian, 2018. "The role of Statistics in the era of big data: Crucial, critical and under-valued," Statistics & Probability Letters, Elsevier, vol. 136(C), pages 20-24.
    13. Xian Jin Xie, 2019. "Research Reproducibility and p-value Threshold," Biomedical Journal of Scientific & Technical Research, Biomedical Research Network+, LLC, vol. 22(5), pages 16934-16936, November.
    14. Chatelain, Jean-Bernard & Ralf, Kirsten, 2021. "Inference on time-invariant variables using panel data: A pretest estimator," Economic Modelling, Elsevier, vol. 97(C), pages 157-166.
    15. Karmakar, Bisheswar & Pal, Sucharita & Gopikrishna, Konga & Tiwari, Onkar Nath & Halder, Gopinath, 2022. "Injection of superheated C1 and C3 alcohols in non-edible Pongamia pinnata oil for semi-continuous uncatalyzed biodiesel synthesis," Renewable Energy, Elsevier, vol. 185(C), pages 850-861.
    16. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    17. Li, Yu & Gao, Xiangyun & An, Sufang & Zheng, Huiling & Wu, Tao, 2021. "Network approach to the dynamic transformation characteristics of the joint impacts of gold and oil on copper," Resources Policy, Elsevier, vol. 70(C).
    18. Nicolas Vallois & Dorian Jullien, 2017. "Estimating Rationality in Economics: A History of Statistical Methods in Experimental Economics," Working Papers halshs-01651070, HAL.
    19. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    20. Li, Xinyi & Yao, Runming & Li, Qin & Ding, Yong & Li, Baizhan, 2018. "An object-oriented energy benchmark for the evaluation of the office building stock," Utilities Policy, Elsevier, vol. 51(C), pages 1-11.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:socres:v:21:y:2016:i:2:p:136-147. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.