IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/s9dp6.html
   My bibliography  Save this paper

Sensitivity analysis for publication bias in meta-analyses

Author

Listed:
  • Mathur, Maya B
  • VanderWeele, Tyler

Abstract

We propose sensitivity analyses for publication bias in meta-analyses. We consider a publication process such that "statistically significant" results are more likely to be published than negative or "nonsignificant" results by an unknown ratio, eta. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse-probability weighting and robust estimation that accommodates non-normal population effects, small meta-analyses, and clustering, we develop sensitivity analyses that enable statements such as: "For publication bias to shift the observed point estimate to the null, 'significant' results would need to be at least 30-fold more likely to be published than negative or 'nonsignificant' results." Comparable statements can be made regarding shifting to a chosen non-null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of eta across disciplines. We show that a worst-case meta-analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta-analysis of only the negative and "nonsignificant" studies; this method sometimes indicates that no amount of such publication bias could "explain away" the results. We illustrate the proposed methods using real-life meta-analyses and provide an R package, PublicationBias.

Suggested Citation

  • Mathur, Maya B & VanderWeele, Tyler, 2019. "Sensitivity analysis for publication bias in meta-analyses," OSF Preprints s9dp6, Center for Open Science.
  • Handle: RePEc:osf:osfxxx:s9dp6
    DOI: 10.31219/osf.io/s9dp6
    as

    Download full text from publisher

    File URL: https://osf.io/download/5c9e7104653f06001834c1b3/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/s9dp6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wolfgang Viechtbauer, 2005. "Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model," Journal of Educational and Behavioral Statistics, , vol. 30(3), pages 261-293, September.
    2. Wooldridge, Jeffrey M., 2007. "Inverse probability weighted estimation for general missing data problems," Journal of Econometrics, Elsevier, vol. 141(2), pages 1281-1301, December.
    3. Robert G. Orwin, 1983. "A Fail-SafeN for Effect Size in Meta-Analysis," Journal of Educational and Behavioral Statistics, , vol. 8(2), pages 157-159, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ivan Ropovik & Matus Adamkovic & David Greger, 2021. "Neglect of publication bias compromises meta-analyses of educational research," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-14, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Turner, Alex J. & Fichera, Eleonora & Sutton, Matt, 2021. "The effects of in-utero exposure to influenza on mental health and mortality risk throughout the life-course," Economics & Human Biology, Elsevier, vol. 43(C).
    2. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    3. Feld, Jan & Salamanca, Nicolás & Zölitz, Ulf, 2019. "Students are almost as effective as professors in university teaching," Economics of Education Review, Elsevier, vol. 73(C).
    4. Hector Espinoza & Stefan Speckesser, 2019. "A Comparison of Earnings Related to Higher Level Vocational/Technical and Academic Education," National Institute of Economic and Social Research (NIESR) Discussion Papers 502, National Institute of Economic and Social Research.
    5. Rashid Javed & Mazhar Mughal, 2019. "Have a Son, Gain a Voice: Son Preference and Female Participation in Household Decision Making," Journal of Development Studies, Taylor & Francis Journals, vol. 55(12), pages 2526-2548, December.
    6. Bradley Hardy & Timothy Smeeding & James P. Ziliak, 2018. "The Changing Safety Net for Low-Income Parents and Their Children: Structural or Cyclical Changes in Income Support Policy?," Demography, Springer;Population Association of America (PAA), vol. 55(1), pages 189-221, February.
    7. Stella Min & Miles G. Taylor, 2018. "Racial and Ethnic Variation in the Relationship Between Student Loan Debt and the Transition to First Birth," Demography, Springer;Population Association of America (PAA), vol. 55(1), pages 165-188, February.
    8. Jain, Apoorva & Peter, Klara Sabirianova, 2017. "Limits to Wage Growth: Understanding the Wage Divergence between Immigrants and Natives," IZA Discussion Papers 10891, Institute of Labor Economics (IZA).
    9. Takeshima, Hiroyuki & Hatzenbuehler, Patrick L. & Edeh, Hyacinth O., 2020. "Effects of agricultural mechanization on economies of scope in crop production in Nigeria," Agricultural Systems, Elsevier, vol. 177(C).
    10. Franziska Gassmann & Bruno Martorano & Jennifer Waidler, 2022. "How Social Assistance Affects Subjective Wellbeing: Lessons from Kyrgyzstan," Journal of Development Studies, Taylor & Francis Journals, vol. 58(4), pages 827-847, April.
    11. Meyer, Bruce D. & Mittag, Nikolas, 2019. "Combining Administrative and Survey Data to Improve Income Measurement," IZA Discussion Papers 12266, Institute of Labor Economics (IZA).
    12. McGovern, Mark E. & Canning, David & Bärnighausen, Till, 2018. "Accounting for non-response bias using participation incentives and survey design: An application using gift vouchers," Economics Letters, Elsevier, vol. 171(C), pages 239-244.
    13. Cantarella, Michele & Strozzi, Chiara, 2019. "Workers in the Crowd: The Labour Market Impact of the Online Platform Economy," IZA Discussion Papers 12327, Institute of Labor Economics (IZA).
    14. Julian Freitag & Anna Kerkhof & Johannes Münster, 2021. "Selective sharing of news items and the political position of news outlets," ECONtribute Discussion Papers Series 056, University of Bonn and University of Cologne, Germany.
    15. Feld, Jan & Zölitz, Ulf, 2022. "The effect of higher-achieving peers on major choices and labor market outcomes," Journal of Economic Behavior & Organization, Elsevier, vol. 196(C), pages 200-219.
    16. Bang, Minji & Gao, Wayne Yuan & Postlewaite, Andrew & Sieg, Holger, 2023. "Using monotonicity restrictions to identify models with partially latent covariates," Journal of Econometrics, Elsevier, vol. 235(2), pages 892-921.
    17. Lu, Yi & Tao, Zhigang, 2009. "Contract enforcement and family control of business: Evidence from China," Journal of Comparative Economics, Elsevier, vol. 37(4), pages 597-609, December.
    18. Garbero, Alessandra & Songsermsawas, Tisorn, 2016. "Impact of modern irrigation on household production and welfare outcomes: Evidence from the PASIDP project in Ethiopia," 2016 Annual Meeting, July 31-August 2, Boston, Massachusetts 235949, Agricultural and Applied Economics Association.
    19. Jorge Luis Garcia & Sonia Digiannatale & Victor G Carreon, 2015. "Why Do Formal Credit, Informal Credit, and both Types of Credits Coexist as Consumer Choices?," Economics Bulletin, AccessEcon, vol. 35(1), pages 89-96.
    20. Yamauchi, Futoshi & Liu, Yanyan, 2011. "Girls take over: Long-term impacts of an early stage education intervention in the Philippines," IFPRI discussion papers 1144, International Food Policy Research Institute (IFPRI).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:s9dp6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.