IDEAS home Printed from https://ideas.repec.org/a/zbw/ireejl/206820.html
   My bibliography  Save this article

Meta-analysis and publication bias: How well does the FAT-PET-PEESE procedure work? A replication study of Alinaghi & Reed (Research Synthesis Methods, 2018)

Author

Listed:
  • Hong, Sanghyun

Abstract

A meta-analysis is a tool for aggregating estimates of a similar "effect" across many studies. Publication bias is the phenomenon where literature is sample selected in favor of studies having statistically significant results and/or having estimates that satisfy pre-conceived expectations. A popular procedure used for conducting meta-analyses in the presence of publication bias is the FAT-PET-PEESE (FPP) procedure. In a recent paper published in Research Synthesis Methods, Alinaghi and Reed (2018), utilizing Monte Carlo simulations, report that the FPP procedure does not work well when used in 'realistic' data environments where true effects differ both across and within studies. AR's findings are important because the FPP approach is dominant in the economics meta-analysis literature. I replicate their results and discover two mistakes, which I subsequently correct. The first mistake is found in a descriptive statistics table, misrepresenting the overview of simulated dataset. The second is associated with the fixed effect estimation, generating erroneous estimated effects and Type I error. Further, I extend their analysis by making their simulation environment even more realistic. Despite producing somewhat different results, my replications generally confirm AR's conclusions about the unreliability of the FPP procedure in realistic data environments.

Suggested Citation

  • Hong, Sanghyun, 2019. "Meta-analysis and publication bias: How well does the FAT-PET-PEESE procedure work? A replication study of Alinaghi & Reed (Research Synthesis Methods, 2018)," International Journal for Re-Views in Empirical Economics (IREE), ZBW - Leibniz Information Centre for Economics, vol. 3(2019-4), pages 1-22.
    • Handle: RePEc:zbw:ireejl:206820
    DOI: 10.18718/81781.13
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/206820/1/IREE-2019-04.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.18718/81781.13?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    2. Peter Nijkamp & Jacques Poot, 2005. "The Last Word on the Wage Curve?," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 421-450, July.
    3. Randolph Luca Bruno & Maria Cipollina, 2018. "A meta†analysis of the indirect impact of foreign direct investment in old and new EU member states: Understanding productivity spillovers," The World Economy, Wiley Blackwell, vol. 41(5), pages 1342-1377, May.
    4. Ángel Melguizo & José González-Páramo, 2013. "Who bears labour taxes and social contributions? A meta-analysis approach," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 4(3), pages 247-271, August.
    5. Megan Linde Leonard & T. D. Stanley & Hristos Doucouliagos, 2014. "Does the UK Minimum Wage Reduce Employment? A Meta-Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 52(3), pages 499-520, September.
    6. Tomáš Havránek, 2015. "Measuring Intertemporal Substitution: The Importance Of Method Choices And Selective Reporting," Journal of the European Economic Association, European Economic Association, vol. 13(6), pages 1180-1204, December.
    7. Havranek, Tomas & Irsova, Zuzana & Janda, Karel, 2012. "Demand for gasoline is more price-inelastic than commonly thought," Energy Economics, Elsevier, vol. 34(1), pages 201-207.
    8. Tomas Havranek & Zuzana Irsova & Tomas Vlach, 2018. "Measuring the Income Elasticity of Water Demand: The Importance of Publication and Endogeneity Biases," Land Economics, University of Wisconsin Press, vol. 94(2), pages 259-283.
    9. Mekbib Gebretsadik Haile & Geoff Pugh, 2013. "Does exchange rate volatility discourage international trade? A meta-regression analysis," The Journal of International Trade & Economic Development, Taylor & Francis Journals, vol. 22(3), pages 321-350, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Reed, W. Robert, 2019. "Meta-analysis and publication bias: How well does the FAT-PET-PEESE procedure work? A reply to Hong (International Journal for Re-Views in Empirical Economics, 2019)," International Journal for Re-Views in Empirical Economics (IREE), ZBW - Leibniz Information Centre for Economics, vol. 3(2019-5), pages 1-4.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:wly:econjl:v::y:2017:i:605:p:f236-f265 is not listed on IDEAS
    2. John P. A. Ioannidis & T. D. Stanley & Hristos Doucouliagos, 2017. "The Power of Bias in Economics Research," Economic Journal, Royal Economic Society, vol. 127(605), pages 236-265, October.
    3. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    4. Slichter, David & Tran, Nhan, 2023. "Do better journals publish better estimates?," MPRA Paper 118433, University Library of Munich, Germany.
    5. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    6. Zigraiova, Diana & Havranek, Tomas & Irsova, Zuzana & Novak, Jiri, 2021. "How puzzling is the forward premium puzzle? A meta-analysis," European Economic Review, Elsevier, vol. 134(C).
    7. Petr Polák, 2019. "The Euro'S Trade Effect: A Meta‐Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 33(1), pages 101-124, February.
    8. Mojmir Hampl & Tomas Havranek, 2020. "Central Bank Equity as an Instrument of Monetary Policy," Comparative Economic Studies, Palgrave Macmillan;Association for Comparative Economic Studies, vol. 62(1), pages 49-68, March.
    9. Fan Yang & Tomas Havranek & Zuzana Irsova & Jiri Novak, 2024. "Is research on hedge fund performance published selectively? A quantitative survey," Journal of Economic Surveys, Wiley Blackwell, vol. 38(4), pages 1085-1131, September.
    10. Ali Elminejad & Tomas Havranek & Roman Horvath & Zuzana Irsova, 2023. "Intertemporal Substitution in Labor Supply: A Meta-Analysis," Review of Economic Dynamics, Elsevier for the Society for Economic Dynamics, vol. 51, pages 1095-1113, December.
    11. Bajzik, Josef & Havranek, Tomas & Irsova, Zuzana & Schwarz, Jiri, 2020. "Estimating the Armington elasticity: The importance of study design and publication bias," Journal of International Economics, Elsevier, vol. 127(C).
    12. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    13. Havranek, Tomas & Irsova, Zuzana & Zeynalova, Olesia, 2017. "Tuition Reduces Enrollment Less Than Commonly Thought," MPRA Paper 78813, University Library of Munich, Germany.
    14. Burke, Paul J. & Yang, Hewen, 2016. "The price and income elasticities of natural gas demand: International evidence," Energy Economics, Elsevier, vol. 59(C), pages 466-474.
    15. Chris Doucouliagos & Jakob de Haan & Jan-Egbert Sturm, 2022. "What drives financial development? A Meta-regression analysis [A new database of financial reforms]," Oxford Economic Papers, Oxford University Press, vol. 74(3), pages 840-868.
    16. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    17. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    18. Kudela, Peter & Havranek, Tomas & Herman, Dominik & Irsova, Zuzana, 2020. "Does daylight saving time save electricity? Evidence from Slovakia," Energy Policy, Elsevier, vol. 137(C).
    19. Tomas Havranek & Zuzana Irsova & Lubica Laslopova & Olesia Zeynalova, 2020. "Skilled and Unskilled Labor Are Less Substitutable than Commonly Thought," Working Papers IES 2020/29, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    20. Havranek, Tomas & Irsova, Zuzana & Janda, Karel & Zilberman, David, 2015. "Selective reporting and the social cost of carbon," Energy Economics, Elsevier, vol. 51(C), pages 394-406.
    21. Anton Astakhov & Tomas Havranek & Jiri Novak, 2019. "Firm Size And Stock Returns: A Quantitative Survey," Journal of Economic Surveys, Wiley Blackwell, vol. 33(5), pages 1463-1492, December.

    More about this item

    Keywords

    Meta-Analysis; Publication Bias; Funnel Asymmetry Test (FAT); Precision Effect Estimate with Standard Error (PEESE); Monte Carlo Simulations; Replication Study;
    All these keywords.

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:ireejl:206820. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.