IDEAS home Printed from https://ideas.repec.org/a/taf/amstat/v73y2019is1p262-270.html
   My bibliography  Save this article

Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication

Author

Listed:
  • Valentin Amrhein
  • David Trafimow
  • Sander Greenland

Abstract

Statistical inference often fails to replicate. One reason is that many results may be selected for drawing inference because some threshold of a statistic like the P-value was crossed, leading to biased reported effect sizes. Nonetheless, considerable non-replication is to be expected even without selective reporting, and generalizations from single studies are rarely if ever warranted. Honestly reported results must vary from replication to replication because of varying assumption violations and random variation; excessive agreement itself would suggest deeper problems, such as failure to publish results in conflict with group expectations or desires. A general perception of a “replication crisis” may thus reflect failure to recognize that statistical tests not only test hypotheses, but countless assumptions and the entire environment in which research takes place. Because of all the uncertain and unknown assumptions that underpin statistical inferences, we should treat inferential statistics as highly unstable local descriptions of relations between assumptions and data, rather than as providing generalizable inferences about hypotheses or models. And that means we should treat statistical results as being much more incomplete and uncertain than is currently the norm. Acknowledging this uncertainty could help reduce the allure of selective reporting: Since a small P-value could be large in a replication study, and a large P-value could be small, there is simply no need to selectively report studies based on statistical results. Rather than focusing our study reports on uncertain conclusions, we should thus focus on describing accurately how the study was conducted, what problems occurred, what data were obtained, what analysis methods were used and why, and what output those methods produced.

Suggested Citation

  • Valentin Amrhein & David Trafimow & Sander Greenland, 2019. "Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 262-270, March.
  • Handle: RePEc:taf:amstat:v:73:y:2019:i:s1:p:262-270
    DOI: 10.1080/00031305.2018.1543137
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1080/00031305.2018.1543137
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1080/00031305.2018.1543137?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Keith R Lohse & Kristin L Sainani & J Andrew Taylor & Michael L Butson & Emma J Knight & Andrew J Vickers, 2020. "Systematic review of the use of “magnitude-based inference” in sports science and medicine," PLOS ONE, Public Library of Science, vol. 15(6), pages 1-22, June.
    2. Markku Maula & Wouter Stam, 2020. "Enhancing Rigor in Quantitative Entrepreneurship Research," Entrepreneurship Theory and Practice, , vol. 44(6), pages 1059-1090, November.
    3. Beecham, Roger & Lovelace, Robin, 2022. "A framework for inserting visually-supported inferences into geographical analysis workflow: application to road safety research," OSF Preprints mfja8, Center for Open Science.
    4. Patrick Vu, 2022. "Can the Replication Rate Tell Us About Publication Bias?," Papers 2206.15023, arXiv.org, revised Jul 2022.
    5. Sander Greenland, 2023. "Divergence versus decision P‐values: A distinction worth making in theory and keeping in practice: Or, how divergence P‐values measure evidence even when decision P‐values do not," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 50(1), pages 54-88, March.
    6. Vu, Patrick, 2022. "Can the Replication Rate Tell Us About Selective Publication?," I4R Discussion Paper Series 3, The Institute for Replication (I4R).
    7. G. Christopher Crawford & Vitaliy Skorodziyevskiy & Casey J. Frid & Thomas E. Nelson & Zahra Booyavi & Diana M. Hechavarria & Xuanye Li & Paul D. Reynolds & Ehsan Teymourian, 2022. "Advancing Entrepreneurship Theory Through Replication: A Case Study on Contemporary Methodological Challenges, Future Best Practices, and an Entreaty for Communality," Entrepreneurship Theory and Practice, , vol. 46(3), pages 779-799, May.
    8. Wilcox, Rand R. & Rousselet, Guillaume A, 2024. "More Reasons Why Replication Is A Difficult Issue," OSF Preprints 9amhe, Center for Open Science.
    9. Austin Chia & Margaret L. Kern, 2021. "Subjective Wellbeing and the Social Responsibilities of Business: an Exploratory Investigation of Australian Perspectives," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 16(5), pages 1881-1908, October.
    10. Wang, Feipeng & Wong, Wing-Keung & Wang, Zheng & Albasher, Gadah & Alsultan, Nouf & Fatemah, Ambreen, 2023. "Emerging pathways to sustainable economic development: An interdisciplinary exploration of resource efficiency, technological innovation, and ecosystem resilience in resource-rich regions," Resources Policy, Elsevier, vol. 85(PA).
    11. Elise S. W. Hung, 2020. "Psychological Risk Factors of Future Drug Offending among Young Offenders in Hong Kong - A Longitudinal Study," International Journal of Psychological Studies, Canadian Center of Science and Education, vol. 12(4), pages 1-31, December.
    12. Bagilet, Vincent & Zabrocki-Hallak, Léo, 2022. "Why Some Acute Health Effects of Air Pollution Could Be Inflated," I4R Discussion Paper Series 11, The Institute for Replication (I4R).
    13. Krzysztof Jajuga & Józef Pociecha & Mirosław Szreder, 2024. "Statistical inference and statistical learning in economic research – selected challenges," Ekonomista, Polskie Towarzystwo Ekonomiczne, issue 2, pages 138-154.
    14. Pablo Martínez-Camblor, 2022. "Learning the Treatment Impact on Time-to-Event Outcomes: The Transcarotid Artery Revascularization Simulated Cohort," IJERPH, MDPI, vol. 19(19), pages 1-12, September.
    15. Sadri, Arash, 2022. "The Ultimate Cause of the “Reproducibility Crisis”: Reductionist Statistics," MetaArXiv yxba5, Center for Open Science.
    16. David Trafimow, 2019. "A Frequentist Alternative to Significance Testing, p -Values, and Confidence Intervals," Econometrics, MDPI, vol. 7(2), pages 1-14, June.
    17. Lippmann, Quentin, 2021. "Are gender quotas on candidates bound to be ineffective?," Journal of Economic Behavior & Organization, Elsevier, vol. 191(C), pages 661-678.
    18. Leon C Reteig & Lionel A Newman & K Richard Ridderinkhof & Heleen A Slagter, 2022. "Effects of tDCS on the attentional blink revisited: A statistical evaluation of a replication attempt," PLOS ONE, Public Library of Science, vol. 17(1), pages 1-23, January.
    19. Arjen Witteloostuijn, 2020. "New-day statistical thinking: A bold proposal for a radical change in practices," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 51(2), pages 274-278, March.
    20. Scoggins, Bermond & Robertson, Matthew P., 2023. "Measuring Transparency in the Social Sciences: Political Science and International Relations," I4R Discussion Paper Series 14, The Institute for Replication (I4R).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:amstat:v:73:y:2019:i:s1:p:262-270. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Longhurst (email available below). General contact details of provider: http://www.tandfonline.com/UTAS20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.