IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/s9dp6.html
   My bibliography  Save this paper

Sensitivity analysis for publication bias in meta-analyses

Author

Listed:
  • Mathur, Maya B
  • VanderWeele, Tyler

Abstract

We propose sensitivity analyses for publication bias in meta-analyses. We consider a publication process such that "statistically significant" results are more likely to be published than negative or "nonsignificant" results by an unknown ratio, eta. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse-probability weighting and robust estimation that accommodates non-normal population effects, small meta-analyses, and clustering, we develop sensitivity analyses that enable statements such as: "For publication bias to shift the observed point estimate to the null, 'significant' results would need to be at least 30-fold more likely to be published than negative or 'nonsignificant' results." Comparable statements can be made regarding shifting to a chosen non-null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of eta across disciplines. We show that a worst-case meta-analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta-analysis of only the negative and "nonsignificant" studies; this method sometimes indicates that no amount of such publication bias could "explain away" the results. We illustrate the proposed methods using real-life meta-analyses and provide an R package, PublicationBias.

Suggested Citation

  • Mathur, Maya B & VanderWeele, Tyler, 2019. "Sensitivity analysis for publication bias in meta-analyses," OSF Preprints s9dp6, Center for Open Science.
  • Handle: RePEc:osf:osfxxx:s9dp6
    DOI: 10.31219/osf.io/s9dp6
    as

    Download full text from publisher

    File URL: https://osf.io/download/5c9e7104653f06001834c1b3/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/s9dp6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert G. Orwin, 1983. "A Fail-SafeN for Effect Size in Meta-Analysis," Journal of Educational and Behavioral Statistics, , vol. 8(2), pages 157-159, June.
    2. Wolfgang Viechtbauer, 2005. "Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model," Journal of Educational and Behavioral Statistics, , vol. 30(3), pages 261-293, September.
    3. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ivan Ropovik & Matus Adamkovic & David Greger, 2021. "Neglect of publication bias compromises meta-analyses of educational research," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-14, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Turner, Alex J. & Fichera, Eleonora & Sutton, Matt, 2021. "The effects of in-utero exposure to influenza on mental health and mortality risk throughout the life-course," Economics & Human Biology, Elsevier, vol. 43(C).
    2. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    3. Michele Cantarella & Chiara Strozzi, 2021. "Workers in the crowd: the labor market impact of the online platform economy [An evaluation of instrumental variable strategies for estimating the effects of catholic schooling]," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 30(6), pages 1429-1458.
    4. Ruoxuan Xiong & Allison Koenecke & Michael Powell & Zhu Shen & Joshua T. Vogelstein & Susan Athey, 2021. "Federated Causal Inference in Heterogeneous Observational Data," Papers 2107.11732, arXiv.org, revised Apr 2023.
    5. N'dri, Lasme Mathieu & Kakinaka, Makoto, 2020. "Financial inclusion, mobile money, and individual welfare: The case of Burkina Faso," Telecommunications Policy, Elsevier, vol. 44(3).
    6. Mittag, Nikolas, 2016. "Correcting for Misreporting of Government Benefits," IZA Discussion Papers 10266, Institute of Labor Economics (IZA).
    7. Feld, Jan & Salamanca, Nicolás & Zölitz, Ulf, 2019. "Students are almost as effective as professors in university teaching," Economics of Education Review, Elsevier, vol. 73(C).
    8. Hector Espinoza & Stefan Speckesser, 2019. "A Comparison of Earnings Related to Higher Level Vocational/Technical and Academic Education," National Institute of Economic and Social Research (NIESR) Discussion Papers 502, National Institute of Economic and Social Research.
    9. Rashid Javed & Mazhar Mughal, 2019. "Have a Son, Gain a Voice: Son Preference and Female Participation in Household Decision Making," Journal of Development Studies, Taylor & Francis Journals, vol. 55(12), pages 2526-2548, December.
    10. Bradley Hardy & Timothy Smeeding & James P. Ziliak, 2018. "The Changing Safety Net for Low-Income Parents and Their Children: Structural or Cyclical Changes in Income Support Policy?," Demography, Springer;Population Association of America (PAA), vol. 55(1), pages 189-221, February.
    11. Stella Min & Miles G. Taylor, 2018. "Racial and Ethnic Variation in the Relationship Between Student Loan Debt and the Transition to First Birth," Demography, Springer;Population Association of America (PAA), vol. 55(1), pages 165-188, February.
    12. Defever, F. & Riaño, A. & Varela, G., 2020. "Evaluating the Impact of Export Finance Support On Firm-level Export Performance: Evidence from Pakistan," Working Papers 20/14, Department of Economics, City University London.
    13. Jain, Apoorva & Peter, Klara Sabirianova, 2017. "Limits to Wage Growth: Understanding the Wage Divergence between Immigrants and Natives," IZA Discussion Papers 10891, Institute of Labor Economics (IZA).
    14. Takeshima, Hiroyuki & Hatzenbuehler, Patrick L. & Edeh, Hyacinth O., 2020. "Effects of agricultural mechanization on economies of scope in crop production in Nigeria," Agricultural Systems, Elsevier, vol. 177(C).
    15. Cai, Tingting & Li, Jianbo & Zhou, Qin & Yin, Songlou & Zhang, Riquan, 2024. "Subgroup detection based on partially linear additive individualized model with missing data in response," Computational Statistics & Data Analysis, Elsevier, vol. 192(C).
    16. Martin Huber, 2012. "Identification of Average Treatment Effects in Social Experiments Under Alternative Forms of Attrition," Journal of Educational and Behavioral Statistics, , vol. 37(3), pages 443-474, June.
    17. Franziska Gassmann & Bruno Martorano & Jennifer Waidler, 2022. "How Social Assistance Affects Subjective Wellbeing: Lessons from Kyrgyzstan," Journal of Development Studies, Taylor & Francis Journals, vol. 58(4), pages 827-847, April.
    18. Meyer, Bruce D. & Mittag, Nikolas, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," IZA Discussion Papers 12266, Institute of Labor Economics (IZA).
    19. Bruce D. Meyer & Derek Wu & Victoria R. Mooers & Carla Medalia, 2019. "The use and misuse of income data and extreme poverty in the United States," AEI Economics Working Papers 1018925, American Enterprise Institute.
    20. Faruque As Sunny & Linlin Fu & Md Sadique Rahman & Zuhui Huang, 2022. "Determinants and Impact of Solar Irrigation Facility (SIF) Adoption: A Case Study in Northern Bangladesh," Energies, MDPI, vol. 15(7), pages 1-17, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:s9dp6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.