IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/yxba5.html
   My bibliography  Save this paper

The Ultimate Cause of the “Reproducibility Crisis”: Reductionist Statistics

Author

Listed:
  • Sadri, Arash

Abstract

Resolving the "replication crisis" is a top priority of the scientific community now. "Reproducibility" is claimed as a central tenet of science and the estimated economic and social burden is huge. Numerous proposals have been made. Still, there lacks not only an established solution but even an agreement on whether there exists a "crisis" or not. Here, by questioning the philosophical foundations of our study designs and analyses, I trace back the "crisis" to reductionist ontologies and methodologies ingrained in the modern statistical methods which have dominated biological, medical, psychological, and social sciences for a century. The crisis is not our inability to "reproduce" results but that we expect to be able to "reproduce" results despite neglecting almost all individual-level and contextual variables of complex processes.

Suggested Citation

  • Sadri, Arash, 2022. "The Ultimate Cause of the “Reproducibility Crisis”: Reductionist Statistics," MetaArXiv yxba5, Center for Open Science.
    • Handle: RePEc:osf:metaar:yxba5
    DOI: 10.31219/osf.io/yxba5
    as

    Download full text from publisher

    File URL: https://osf.io/download/6266a2a82aff521d7cb6a9b6/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/yxba5?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Alessandro Selvitella, 2017. "The ubiquity of the Simpson’s Paradox," Journal of Statistical Distributions and Applications, Springer, vol. 4(1), pages 1-16, December.
    2. David Colquhoun, 2019. "The False Positive Risk: A Proposal Concerning What to Do About p-Values," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 192-201, March.
    3. Valentin Amrhein & David Trafimow & Sander Greenland, 2019. "Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 262-270, March.
    4. Bernhard Voelkl & Lucile Vogt & Emily S Sena & Hanno Würbel, 2018. "Reproducibility of preclinical animal research improves with heterogeneity of study samples," PLOS Biology, Public Library of Science, vol. 16(2), pages 1-13, February.
    5. Blakeley B. McShane & Jennifer L. Tackett & Ulf Böckenholt & Andrew Gelman, 2019. "Large-Scale Replication Projects in Contemporary Psychological Research," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 99-105, March.
    6. Leonid Hanin, 2021. "Cavalier Use of Inferential Statistics Is a Major Source of False and Irreproducible Scientific Findings," Mathematics, MDPI, vol. 9(6), pages 1-13, March.
    7. Takuji Usui & Malcolm R Macleod & Sarah K McCann & Alistair M Senior & Shinichi Nakagawa, 2021. "Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research," PLOS Biology, Public Library of Science, vol. 19(5), pages 1-20, May.
    8. Cartwright,Nancy, 1999. "The Dappled World," Cambridge Books, Cambridge University Press, number 9780521643368, January.
    9. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    10. Raymond Hubbard & Brian D. Haig & Rahul A. Parsa, 2019. "The Limited Role of Formal Statistical Inference in Scientific Inference," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 91-98, March.
    11. Cartwright,Nancy, 1999. "The Dappled World," Cambridge Books, Cambridge University Press, number 9780521644112, January.
    12. Scott Marek & Brenden Tervo-Clemmens & Finnegan J. Calabro & David F. Montez & Benjamin P. Kay & Alexander S. Hatoum & Meghan Rose Donohue & William Foran & Ryland L. Miller & Timothy J. Hendrickson &, 2022. "Reproducible brain-wide association studies require thousands of individuals," Nature, Nature, vol. 603(7902), pages 654-660, March.
    13. Shareen A Iqbal & Joshua D Wallach & Muin J Khoury & Sheri D Schully & John P A Ioannidis, 2016. "Reproducible Research Practices and Transparency across the Biomedical Literature," PLOS Biology, Public Library of Science, vol. 14(1), pages 1-13, January.
    14. Blakeley B. McShane & David Gal, 2017. "Rejoinder: Statistical Significance and the Dichotomization of Evidence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 904-908, July.
    15. Nicole C Nelson & Kelsey Ichikawa & Julie Chung & Momin M Malik, 2021. "Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis," PLOS ONE, Public Library of Science, vol. 16(7), pages 1-20, July.
    16. Leonard P Freedman & Iain M Cockburn & Timothy S Simcoe, 2015. "The Economics of Reproducibility in Preclinical Research," PLOS Biology, Public Library of Science, vol. 13(6), pages 1-9, June.
    17. Rotem Botvinik-Nezer & Felix Holzmeister & Colin F. Camerer & Anna Dreber & Juergen Huber & Magnus Johannesson & Michael Kirchler & Roni Iwanir & Jeanette A. Mumford & R. Alison Adcock & Paolo Avesani, 2020. "Variability in the analysis of a single neuroimaging dataset by many teams," Nature, Nature, vol. 582(7810), pages 84-88, June.
    18. Matthew J Page & Larissa Shamseer & Douglas G Altman & Jennifer Tetzlaff & Margaret Sampson & Andrea C Tricco & Ferrán Catalá-López & Lun Li & Emma K Reid & Rafael Sarkis-Onofre & David Moher, 2016. "Epidemiology and Reporting Characteristics of Systematic Reviews of Biomedical Research: A Cross-Sectional Study," PLOS Medicine, Public Library of Science, vol. 13(5), pages 1-30, May.
    19. Nosek, Brian A. & Ebersole, Charles R. & DeHaven, Alexander Carl & Mellor, David Thomas, 2018. "The Preregistration Revolution," OSF Preprints 2dxu5, Center for Open Science.
    20. Blakeley B. McShane & David Gal, 2017. "Statistical Significance and the Dichotomization of Evidence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 885-895, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sadri, Arash, 2022. "Machine Learning Can Solve the Reproducibility Crisis by Supplanting Reductionist Statistics," MetaArXiv yxba5_v1, Center for Open Science.
    2. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    3. Christopher Allen & David M A Mehler, 2019. "Open science challenges, benefits and tips in early career and beyond," PLOS Biology, Public Library of Science, vol. 17(5), pages 1-14, May.
    4. Gunter, Ulrich & Önder, Irem & Smeral, Egon, 2019. "Scientific value of econometric tourism demand studies," Annals of Tourism Research, Elsevier, vol. 78(C), pages 1-1.
    5. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    6. Luigi Pace & Alessandra Salvan, 2020. "Likelihood, Replicability and Robbins' Confidence Sequences," International Statistical Review, International Statistical Institute, vol. 88(3), pages 599-615, December.
    7. Glenn Shafer, 2021. "Testing by betting: A strategy for statistical and scientific communication," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 407-431, April.
    8. Strømland, Eirik, 2019. "Preregistration and reproducibility," Journal of Economic Psychology, Elsevier, vol. 75(PA).
    9. Tom Engsted, 2024. "What Is the False Discovery Rate in Empirical Research?," Econ Journal Watch, Econ Journal Watch, vol. 21(1), pages 1-92–112, March.
    10. Maier, Maximilian & VanderWeele, Tyler J. & Mathur, Maya B, 2021. "Using Selection Models to Assess Sensitivity to Publication Bias: A Tutorial and Call for More Routine Use," MetaArXiv tp45u_v1, Center for Open Science.
    11. Julian Reiss, 2001. "Natural economic quantities and their measurement," Journal of Economic Methodology, Taylor & Francis Journals, vol. 8(2), pages 287-311.
    12. Matthew Rosenblatt & Link Tejavibulya & Rongtao Jiang & Stephanie Noble & Dustin Scheinost, 2024. "Data leakage inflates prediction performance in connectome-based machine learning models," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    13. Aumann, Craig A., 2007. "A methodology for developing simulation models of complex systems," Ecological Modelling, Elsevier, vol. 202(3), pages 385-396.
    14. Stephen Pratten, 2007. "Realism, closed systems and abstraction," Journal of Economic Methodology, Taylor & Francis Journals, vol. 14(4), pages 473-497.
    15. Bertoldi, Paolo & Mosconi, Rocco, 2020. "Do energy efficiency policies save energy? A new approach based on energy policy indicators (in the EU Member States)," Energy Policy, Elsevier, vol. 139(C).
    16. Midgley, Gerald, 2008. "Response to paper "Systems thinking" by D. Cabrera et al.:: The unification of systems thinking: Is there gold at the end of the rainbow?," Evaluation and Program Planning, Elsevier, vol. 31(3), pages 317-321, August.
    17. Nicolas Brisset, 2018. "Models as speech acts: the telling case of financial models," Journal of Economic Methodology, Taylor & Francis Journals, vol. 25(1), pages 21-41, January.
    18. Toby Ord & Rafaela Hillerbrand & Anders Sandberg, 2010. "Probing the improbable: methodological challenges for risks with low probabilities and high stakes," Journal of Risk Research, Taylor & Francis Journals, vol. 13(2), pages 191-205, March.
    19. Simon Hall & Nilufa Ali & Nick Chater & Mike Oaksford, 2016. "Discounting and Augmentation in Causal Conditional Reasoning: Causal Models or Shallow Encoding?," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-23, December.
    20. David J. Hand, 2022. "Trustworthiness of statistical inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(1), pages 329-347, January.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:yxba5. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.