IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/jt5zf.html
   My bibliography  Save this paper

Likelihood Ratio Test for Publication Bias – a Proof of Concept

Author

Listed:
  • Lenartowicz, Paweł

Abstract

Publication bias poses a serious challenge to the integrity of scientific research and meta-analyses. There exist persistent methodological obstacles for estimating this bias, especially with heterogeneous dataset, where studies vary widely in methodologies and effect sizes. To address this gap, I propose a Likelihood Ratio Test for Publication Bias, a statistical method designed to detect and quantify publication bias in datasets of heterogeneous studies results. I also show the proof-of-concept implementation developed in Python and simulations that evaluate the performance. The results demonstrate that this new method clearly outperforms existing methods like Z-Curve 2 and the Caliper test in estimating the magnitude of publication bias, showing higher precision and reliability, with still some space for improvement due to spotted errors in the implemented algorithm. While inherent challenges in publication bias detection remain, such as the influence of different research practices and the need for large sample sizes, the Likelihood Ratio Test offers a significant advancement in addressing these issues.

Suggested Citation

  • Lenartowicz, Paweł, 2024. "Likelihood Ratio Test for Publication Bias – a Proof of Concept," MetaArXiv jt5zf, Center for Open Science.
  • Handle: RePEc:osf:metaar:jt5zf
    DOI: 10.31219/osf.io/jt5zf
    as

    Download full text from publisher

    File URL: https://osf.io/download/6734acdfc0a0f4ba7162f507/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/jt5zf?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Gerber, Alan & Malhotra, Neil, 2008. "Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals," Quarterly Journal of Political Science, now publishers, vol. 3(3), pages 313-326, October.
    2. Abel Brodeur & Nikolai M. Cook & Jonathan S. Hartley & Anthony Heyes, 2024. "Do Preregistration and Preanalysis Plans Reduce p-Hacking and Publication Bias? Evidence from 15,992 Test Statistics and Suggestions for Improvement," Journal of Political Economy Microeconomics, University of Chicago Press, vol. 2(3), pages 527-561.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lenartowicz, Paweł, 2024. "Likelihood Ratio Test for Publication Bias – a Proof of Concept," MetaArXiv jt5zf_v1, Center for Open Science.
    2. Auspurg Katrin & Hinz Thomas, 2011. "What Fuels Publication Bias?: Theoretical and Empirical Analyses of Risk Factors Using the Caliper Test," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 231(5-6), pages 636-660, October.
    3. Rose, Julian & Neubauer, Florian & Ankel-Peters, Jörg, 2024. "Long-Term Effects of the Targeting the Ultra-Poor Program - A Reproducibility and Replicability Assessment of Banerjee et al. (2021)," I4R Discussion Paper Series 142, The Institute for Replication (I4R).
    4. Graham Elliott & Nikolay Kudrin & Kaspar Wüthrich, 2022. "Detecting p‐Hacking," Econometrica, Econometric Society, vol. 90(2), pages 887-906, March.
    5. Marcel Fafchamps & Julien Labonne, 2016. "Using Split Samples to Improve Inference about Causal Effects," NBER Working Papers 21842, National Bureau of Economic Research, Inc.
    6. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    7. Fernando Hoces de la Guardia & Sean Grant & Edward Miguel, 2021. "A framework for open policy analysis," Science and Public Policy, Oxford University Press, vol. 48(2), pages 154-163.
    8. Tomas Havranek & Zuzana Irsova & Lubica Laslopova & Olesia Zeynalova, 2020. "Skilled and Unskilled Labor Are Less Substitutable than Commonly Thought," Working Papers IES 2020/29, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    9. Brodeur, Abel & Esterling, Kevin & Ankel-Peters, Jörg & Bueno, Natália S & Desposato, Scott & Dreber, Anna & Genovese, Federica & Green, Donald P & Hepplewhite, Matthew & de la Guardia, Fernando Hoces, 2024. "Promoting Reproducibility and Replicability in Political Science," Department of Economics, Working Paper Series qt23n3n3dg, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    10. Cala, Petr & Havranek, Tomas & Irsova, Zuzana & Matousek, Jindrich & Novak, Jiri, 2022. "Financial Incentives and Performance: A Meta-Analysis of Economics Evidence," EconStor Preprints 265535, ZBW - Leibniz Information Centre for Economics.
    11. Alexander L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin F. Camerer, 2024. "Meta-analysis of Empirical Estimates of Loss Aversion," Journal of Economic Literature, American Economic Association, vol. 62(2), pages 485-516, June.
    12. Havranek, Tomas & Irsova, Zuzana & Laslopova, Lubica & Zeynalova, Olesia, 2020. "The Elasticity of Substitution between Skilled and Unskilled Labor: A Meta-Analysis," MPRA Paper 102598, University Library of Munich, Germany.
    13. Brodeur, Abel & Kattan, Lamis & Musumeci, Marco, 2024. "Job Market Stars," I4R Discussion Paper Series 141, The Institute for Replication (I4R).
    14. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2018. "Methods Matter: P-Hacking and Causal Inference in Economics," IZA Discussion Papers 11796, Institute of Labor Economics (IZA).
    15. Graham Elliott & Nikolay Kudrin & Kaspar Wuthrich, 2022. "The Power of Tests for Detecting $p$-Hacking," Papers 2205.07950, arXiv.org, revised Apr 2024.
    16. Dominika Ehrenbergerova & Josef Bajzik, 2020. "The Effect of Monetary Policy on House Prices - How Strong is the Transmission?," Working Papers 2020/14, Czech National Bank.
    17. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    18. Bajzík, Josef & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2023. "Does Shareholder Activism Create Value? A Meta-Analysis," CEPR Discussion Papers 18233, C.E.P.R. Discussion Papers.
    19. Felix Chopra & Ingar Haaland & Christopher Roth & Andreas Stegmann, 2024. "The Null Result Penalty," The Economic Journal, Royal Economic Society, vol. 134(657), pages 193-219.
    20. Beger, Andreas & Dorff, Cassy L. & Ward, Michael D., 2016. "Irregular leadership changes in 2014: Forecasts using ensemble, split-population duration models," International Journal of Forecasting, Elsevier, vol. 32(1), pages 98-111.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:jt5zf. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.