IDEAS home Printed from https://ideas.repec.org/a/bla/biomet/v79y2023i1p319-331.html
   My bibliography  Save this article

Modelling publication bias and p‐hacking

Author

Listed:
  • Jonas Moss
  • Riccardo De Bin

Abstract

Publication bias and p‐hacking are two well‐known phenomena that strongly affect the scientific literature and cause severe problems in meta‐analyses. Due to these phenomena, the assumptions of meta‐analyses are seriously violated and the results of the studies cannot be trusted. While publication bias is very often captured well by the weighting function selection model, p‐hacking is much harder to model and no definitive solution has been found yet. In this paper, we advocate the selection model approach to model publication bias and propose a mixture model for p‐hacking. We derive some properties for these models, and we compare them formally and through simulations. Finally, two real data examples are used to show how the models work in practice.

Suggested Citation

  • Jonas Moss & Riccardo De Bin, 2023. "Modelling publication bias and p‐hacking," Biometrics, The International Biometric Society, vol. 79(1), pages 319-331, March.
    • Handle: RePEc:bla:biomet:v:79:y:2023:i:1:p:319-331
    DOI: 10.1111/biom.13560
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/biom.13560
    Download Restriction: no
    File URL: https://libkey.io/10.1111/biom.13560?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Carpenter, Bob & Gelman, Andrew & Hoffman, Matthew D. & Lee, Daniel & Goodrich, Ben & Betancourt, Michael & Brubaker, Marcus & Guo, Jiqiang & Li, Peter & Riddell, Allen, 2017. "Stan: A Probabilistic Programming Language," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 76(i01).
    2. Jack Vevea & Larry Hedges, 1995. "A general linear model for estimating effect size in the presence of publication bias," Psychometrika, Springer;The Psychometric Society, vol. 60(3), pages 419-435, September.
    3. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ferrero, Marta & Vadillo, Miguel A. & León, Samuel P., 2021. "A valid evaluation of the theory of multiple intelligences is not yet possible: Problems of methodological quality for intervention studies," Intelligence, Elsevier, vol. 88(C).
    2. Irsova, Zuzana & Bom, Pedro Ricardo Duarte & Havranek, Tomas & Rachinger, Heiko, 2023. "Spurious Precision in Meta-Analysis," MetaArXiv 3qp2w, Center for Open Science.
    3. Irsova, Zuzana & Doucouliagos, Hristos & Havranek, Tomas & Stanley, T. D., 2023. "Meta-Analysis of Social Science Research: A Practitioner’s Guide," EconStor Preprints 273719, ZBW - Leibniz Information Centre for Economics.
    4. Maximilian Maier & Tyler J. VanderWeele & Maya B. Mathur, 2022. "Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use," Campbell Systematic Reviews, John Wiley & Sons, vol. 18(3), September.
    5. Maya B. Mathur & Tyler J. VanderWeele, 2020. "Sensitivity analysis for publication bias in meta‐analyses," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(5), pages 1091-1119, November.
    6. Sanghyun Hong & W. Robert Reed, 2020. "Using Monte Carlo Experiments to Select Meta-Analytic Estimators," Working Papers in Economics 20/10, University of Canterbury, Department of Economics and Finance.
    7. Maria Abreu & Henri L. F. de Groot & Raymond J. G. M. Florax, 2005. "A Meta‐Analysis of β‐Convergence: the Legendary 2%," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 389-420, July.
    8. Furukawa, Chishio, 2019. "Publication Bias under Aggregation Frictions: Theory, Evidence, and a New Correction Method," EconStor Preprints 194798, ZBW - Leibniz Information Centre for Economics.
    9. Robbie C M van Aert & Jelte M Wicherts & Marcel A L M van Assen, 2019. "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-32, April.
    10. Yucheon Kim & Songyi Lee, 2023. "A Systematic Review and Meta-Analysis of the Effectiveness of Non-Face-to-Face Coaching," Sustainability, MDPI, vol. 15(12), pages 1-15, June.
    11. Carl P. Schmertmann & Marcos R. Gonzaga, 2018. "Bayesian Estimation of Age-Specific Mortality and Life Expectancy for Small Areas With Defective Vital Records," Demography, Springer;Population Association of America (PAA), vol. 55(4), pages 1363-1388, August.
    12. Jose Pina-Sánchez & John Paul Gosling, 2020. "Tackling selection bias in sentencing data analysis: a new approach based on a scale of severity," Quality & Quantity: International Journal of Methodology, Springer, vol. 54(3), pages 1047-1073, June.
    13. Sally Paganin & Christopher J. Paciorek & Claudia Wehrhahn & Abel Rodríguez & Sophia Rabe-Hesketh & Perry de Valpine, 2023. "Computational Strategies and Estimation Performance With Bayesian Semiparametric Item Response Theory Models," Journal of Educational and Behavioral Statistics, , vol. 48(2), pages 147-188, April.
    14. Francis,David C. & Kubinec ,Robert, 2022. "Beyond Political Connections : A Measurement Model Approach to Estimating Firm-levelPolitical Influence in 41 Economies," Policy Research Working Paper Series 10119, The World Bank.
    15. Qian Li & Yan Chen & Shikun Sun & Muyuan Zhu & Jing Xue & Zihan Gao & Jinfeng Zhao & Yihe Tang, 2022. "Research on Crop Irrigation Schedules Under Deficit Irrigation—A Meta-analysis," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(12), pages 4799-4817, September.
    16. Bart Verkuil & Serpil Atasayi & Marc L Molendijk, 2015. "Workplace Bullying and Mental Health: A Meta-Analysis on Cross-Sectional and Longitudinal Data," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-16, August.
    17. Martinovici, A., 2019. "Revealing attention - how eye movements predict brand choice and moment of choice," Other publications TiSEM 7dca38a5-9f78-4aee-bd81-c, Tilburg University, School of Economics and Management.
    18. Damiano Pizzol & Mike Trott & Igor Grabovac & Mario Antunes & Anna Claudia Colangelo & Simona Ippoliti & Cristian Petre Ilie & Anne Carrie & Nicola Veronese & Lee Smith, 2021. "Laparoscopy in Low-Income Countries: 10-Year Experience and Systematic Literature Review," IJERPH, MDPI, vol. 18(11), pages 1-11, May.
    19. Yongping Bao & Ludwig Danwitz & Fabian Dvorak & Sebastian Fehrler & Lars Hornuf & Hsuan Yu Lin & Bettina von Helversen, 2022. "Similarity and Consistency in Algorithm-Guided Exploration," CESifo Working Paper Series 10188, CESifo.
    20. Heinrich, Torsten & Yang, Jangho & Dai, Shuanping, 2020. "Growth, development, and structural change at the firm-level: The example of the PR China," MPRA Paper 105011, University Library of Munich, Germany.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:biomet:v:79:y:2023:i:1:p:319-331. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0006-341X .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.