IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/gmfs9_v1.html
   My bibliography  Save this paper

Why is the statistical revolution not progressing? Vicious cycle of the scientific reform

Author

Listed:
  • Bialek, Michal

    (University of Wroclaw)

  • Misiak, Michał

    (University of Wroclaw)

  • Dziekan, Martyna

Abstract

To analyze the results of the research, behavioral scientists widely use a statistical rule that sets the significance level to 0.05. Recently, two recommendations on how to improve statistical inference were published: to redefine statistical significance to 0.005, and to select and justify the alpha. We analyzed the empirical work that cited the original recommendation papers, as well as the papers published by the scientist that co-authored the publications. About half of the numerous papers citing these recommendations adhered to them already in the first year since their publication. What is striking, the original authors that proposed the recommendations followed their own recommendations only in 6% of their empirical work. We surveyed the authors asking them to identify major obstacles they experienced while trying to implement their own recommendations, and obstacles they think others could expect or experience.

Suggested Citation

  • Bialek, Michal & Misiak, Michał & Dziekan, Martyna, 2021. "Why is the statistical revolution not progressing? Vicious cycle of the scientific reform," OSF Preprints gmfs9_v1, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:gmfs9_v1
    DOI: 10.31219/osf.io/gmfs9_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/614c71d615e9a4004b923c2f/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/gmfs9_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Georgios Gerasimou, 2016. "Asymmetric dominance, deferral, and status quo bias in a behavioral model of choice," Theory and Decision, Springer, vol. 80(2), pages 295-312, February.
    2. Daniel J. Benjamin & James O. Berger & Magnus Johannesson & Brian A. Nosek & E.-J. Wagenmakers & Richard Berk & Kenneth A. Bollen & Björn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Chr, 2018. "Redefine statistical significance," Nature Human Behaviour, Nature, vol. 2(1), pages 6-10, January.
      • Daniel Benjamin & James Berger & Magnus Johannesson & Brian Nosek & E. Wagenmakers & Richard Berk & Kenneth Bollen & Bjorn Brembs & Lawrence Brown & Colin Camerer & David Cesarini & Christopher Chambe, 2017. "Redefine Statistical Significance," Artefactual Field Experiments 00612, The Field Experiments Website.
    3. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    4. Megan L Head & Luke Holman & Rob Lanfear & Andrew T Kahn & Michael D Jennions, 2015. "The Extent and Consequences of P-Hacking in Science," PLOS Biology, Public Library of Science, vol. 13(3), pages 1-15, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    2. Bialek, Michal & Misiak, Michał & Dziekan, Martyna, 2021. "Why is the statistical revolution not progressing? Vicious cycle of the scientific reform," OSF Preprints gmfs9, Center for Open Science.
    3. Abel Brodeur, Nikolai M. Cook, Anthony Heyes, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments," LCERPA Working Papers am0133, Laurier Centre for Economic Research and Policy Analysis.
    4. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.
    5. Rinne, Sonja, 2024. "Estimating the merit-order effect using coarsened exact matching: Reconciling theory with the empirical results to improve policy implications," Energy Policy, Elsevier, vol. 185(C).
    6. Graham Elliott & Nikolay Kudrin & Kaspar Wüthrich, 2022. "Detecting p‐Hacking," Econometrica, Econometric Society, vol. 90(2), pages 887-906, March.
    7. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2022. "We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell us about p-Hacking and Publication Bias in Online Experiments," GLO Discussion Paper Series 1157, Global Labor Organization (GLO).
    8. Anna Dreber & Magnus Johannesson & Yifan Yang, 2024. "Selective reporting of placebo tests in top economics journals," Economic Inquiry, Western Economic Association International, vol. 62(3), pages 921-932, July.
    9. Graham Elliott & Nikolay Kudrin & Kaspar Wuthrich, 2022. "The Power of Tests for Detecting $p$-Hacking," Papers 2205.07950, arXiv.org, revised Apr 2024.
    10. Ron Berman & Christophe Van den Bulte, 2022. "False Discovery in A/B Testing," Management Science, INFORMS, vol. 68(9), pages 6762-6782, September.
    11. Tom Engsted, 2024. "What Is the False Discovery Rate in Empirical Research?," Econ Journal Watch, Econ Journal Watch, vol. 21(1), pages 1-92–112, March.
    12. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    13. Danielle V. Handel & Eric A. Hanushek, 2024. "Contexts of Convenience: Generalizing from Published Evaluations of School Finance Policies," Evaluation Review, , vol. 48(3), pages 461-494, June.
    14. Libman, A., 2024. ""Zoo" of empirical results: Quantitative research and accumulation of knowledge in social sciences," Journal of the New Economic Association, New Economic Association, vol. 65(4), pages 178-194.
    15. Cantone, Giulio Giacomo, 2023. "The multiversal methodology as a remedy of the replication crisis," MetaArXiv kuhmz, Center for Open Science.
    16. David Gonzalez-Jimenez & Francesco Capozza & Thomas Dirkmaat & Evelien van de Veer & Amber van Druten & Aurélien Baillon, 2025. "Falling and failing (to learn) : Evidence from a nation-wide cybersecurity field experiment with SMEs," Post-Print hal-04875787, HAL.
    17. Cantone, Giulio Giacomo, 2023. "The multiversal methodology as a remedy of the replication crisis," MetaArXiv kuhmz_v1, Center for Open Science.
    18. Beau Coker & Cynthia Rudin & Gary King, 2021. "A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results," Management Science, INFORMS, vol. 67(10), pages 6174-6197, October.
    19. Aubry, Amandine & Héricourt, Jérôme & Marchal, Léa & Nedoncelle, Clément, 2022. "Does Immigration AffectWages? A Meta-Analysis," CEPREMAP Working Papers (Docweb) 2202, CEPREMAP.
    20. Dreber, Anna & Heikensten, Emma & Säve-Söderbergh, Jenny, 2022. "Why do women ask for less?," Labour Economics, Elsevier, vol. 78(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:gmfs9_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.