IDEAS home Printed from https://ideas.repec.org/p/jrp/jrpwrp/2013-040.html
   My bibliography  Save this paper

Identifying Genuine Effects in Observational Research by Means of Meta-Regressions

Author

Listed:
  • Stephan B. Bruns

    (Max Planck Institute of Economics, Jena)

Abstract

Meta-regression models are increasingly utilized to integrate empirical results across studies while controlling for the potential threats of data-mining and publication bias. We propose extended meta-regression models and evaluate their performance in identifying genuine em- pirical effects by means of a comprehensive simulation study for various scenarios that are prevalent in empirical economics. We can show that the meta-regression models here pro- posed systematically outperform the prior gold standard of meta-regression analysis of re- gression coefficients. Most meta-regression models are robust to the presence of publication bias, but data-mining bias leads to seriously inflated type I errors and has to be addressed explicitly.

Suggested Citation

  • Stephan B. Bruns, 2013. "Identifying Genuine Effects in Observational Research by Means of Meta-Regressions," Jena Economics Research Papers 2013-040, Friedrich-Schiller-University Jena.
    • Handle: RePEc:jrp:jrpwrp:2013-040
    as

    Download full text from publisher

    File URL: https://oweb.b67.uni-jena.de/Papers/jerp2013/wp_2013_040.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. James H. Stock, 2010. "The Other Transformation in Econometric Practice: Robust Tools for Inference," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 83-94, Spring.
    2. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    3. Stanley, T. D. & Doucouliagos, Chris, 2007. "Identifying and correcting publication selection bias in the efficiency-wage literature: Heckman Meta-Regression," Working Papers eco_2007_11, Deakin University, Department of Economics.
    4. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    5. Frey, Bruno S, 2003. "Publishing as Prostitution?--Choosing between One's Own Ideas and Academic Success," Public Choice, Springer, vol. 116(1-2), pages 205-223, July.
    6. David F. Hendry & Hans-Martin Krolzig, 2005. "The Properties of Automatic "GETS" Modelling," Economic Journal, Royal Economic Society, vol. 115(502), pages 32-61, March.
    7. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    8. Rosenbusch, Nina & Brinckmann, Jan & Müller, Verena, 2013. "Does acquiring venture capital pay off for the funded firms? A meta-analysis on the relationship between venture capital investment and funded firm financial performance," Journal of Business Venturing, Elsevier, vol. 28(3), pages 335-353.
    9. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    10. David Card & Stefano DellaVigna, 2013. "Nine Facts about Top Journals in Economics," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 144-161, March.
    11. Hristos Doucouliagos & T. D. Stanley, 2009. "Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 47(2), pages 406-428, June.
    12. T. D. Stanley, 2008. "Meta‐Regression Methods for Detecting and Estimating Empirical Effects in the Presence of Publication Selection," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(1), pages 103-127, February.
    13. Neal S Young, 2008. "Why Current Publication May Distort Science," Working Papers id:1757, eSocialSciences.
    14. Stephan B. Bruns & Christian Gross & David I. Stern, 2014. "Is There Really Granger Causality between Energy Use and Output?," The Energy Journal, , vol. 35(4), pages 101-134, October.
    15. Neal S Young & John P A Ioannidis & Omar Al-Ubaydli, 2008. "Why Current Publication Practices May Distort Science," PLOS Medicine, Public Library of Science, vol. 5(10), pages 1-5, October.
    16. T. D. Stanley & Stephen B. Jarrell, 2005. "Meta‐Regression Analysis: A Quantitative Method of Literature Surveys," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 299-308, July.
    17. Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-243, May.
    18. Hristos Doucouliagos & Martin Paldam, 2005. "Aid Effectiveness on Growth. A Meta Study," Economics Working Papers 2005-13, Department of Economics and Business Economics, Aarhus University.
    19. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    20. Sala-i-Martin, Xavier, 1997. "I Just Ran Two Million Regressions," American Economic Review, American Economic Association, vol. 87(2), pages 178-183, May.
    21. Doucouliagos, Chris & Stanley, T.D. & Giles, Margaret, 2012. "Are estimates of the value of a statistical life exaggerated?," Journal of Health Economics, Elsevier, vol. 31(1), pages 197-206.
    22. Leamer, Edward E & Leonard, Herman B, 1983. "Reporting the Fragility of Regression Estimates," The Review of Economics and Statistics, MIT Press, vol. 65(2), pages 306-317, May.
    23. T. D. Stanley, 2005. "Integrating the Empirical Tests of the Natural Rate Hypothesis: A Meta‐Regression Analysis," Kyklos, Wiley Blackwell, vol. 58(4), pages 611-634, November.
    24. repec:bla:econom:v:47:y:1980:i:188:p:387-406 is not listed on IDEAS
    25. Sims, Christopher A, 1988. "Uncertainty across Models," American Economic Review, American Economic Association, vol. 78(2), pages 163-167, May.
    26. T. D. Stanley, 2001. "Wheat from Chaff: Meta-analysis as Quantitative Literature Review," Journal of Economic Perspectives, American Economic Association, vol. 15(3), pages 131-150, Summer.
    27. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bouoiyour, Jamal & Selmi, Refk, 2014. "How Does Exchange Rate Uncertainty interact with International Trade? A Meta-Analysis Revisited," MPRA Paper 56201, University Library of Munich, Germany.
    2. Uwe Cantner & Ivan Savin, 2014. "Do Firms Benefit from Complementarity Effect in R&D and What Drives their R&D Strategy Choices?," Jena Economics Research Papers 2014-023, Friedrich-Schiller-University Jena.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. T.D. Stanley, 2013. "Does economics add up? An introduction to meta-regression analysis," European Journal of Economics and Economic Policies: Intervention, Edward Elgar Publishing, vol. 10(2), pages 207-220.
    2. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    3. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    4. Stephan B. Bruns, 2017. "Meta-Regression Models and Observational Research," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 79(5), pages 637-653, October.
    5. Ugur, Mehmet & Trushin, Eshref & Solomon, Edna & Guidi, Francesco, 2016. "R&D and productivity in OECD firms and industries: A hierarchical meta-regression analysis," Research Policy, Elsevier, vol. 45(10), pages 2069-2086.
    6. Petra Valickova & Tomas Havranek & Roman Horvath, 2015. "Financial Development And Economic Growth: A Meta-Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 29(3), pages 506-526, July.
    7. Stephan B. Bruns, 2016. "The Fragility of Meta-Regression Models in Observational Research," MAGKS Papers on Economics 201603, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    8. Tomáš Havránek, 2010. "Rose effect and the euro: is the magic gone?," Review of World Economics (Weltwirtschaftliches Archiv), Springer;Institut für Weltwirtschaft (Kiel Institute for the World Economy), vol. 146(2), pages 241-261, June.
    9. Stanley, T. D. & Doucouliagos, Hristos, 2011. "Meta-regression approximations to reduce publication selection bias," Working Papers eco_2011_4, Deakin University, Department of Economics.
    10. Brodeur, Abel & Cook, Nikolai & Heyes, Anthony, 2018. "Methods Matter: P-Hacking and Causal Inference in Economics," IZA Discussion Papers 11796, Institute of Labor Economics (IZA).
    11. Havranek, Tomas & Irsova, Zuzana & Janda, Karel, 2012. "Demand for gasoline is more price-inelastic than commonly thought," Energy Economics, Elsevier, vol. 34(1), pages 201-207.
    12. Hang, Markus & Geyer-Klingeberg, Jerome & Rathgeber, Andreas W. & Stöckl, Stefan, 2018. "Measurement matters—A meta-study of the determinants of corporate capital structure," The Quarterly Review of Economics and Finance, Elsevier, vol. 68(C), pages 211-225.
    13. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    14. Stéphane Goutte & David Guerreiro & Bilel Sanhaji & Sophie Saglio & Julien Chevallier, 2019. "International Financial Markets," Post-Print halshs-02183053, HAL.
    15. Ugur, Mehmet & Guidi, Francesco & Solomon, Edna & Trushin, Eshref, 2014. "R&D investment, productivity and rates of return: A meta-analysis of the evidence on OECD firms and industries," MPRA Paper 59686, University Library of Munich, Germany, revised 22 Oct 2014.
    16. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    17. Mehmet Ugur, 2014. "Corruption'S Direct Effects On Per-Capita Income Growth: A Meta-Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 28(3), pages 472-490, July.
    18. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.
    19. Tomas Havranek & Zuzana Irsova, 2012. "Survey Article: Publication Bias in the Literature on Foreign Direct Investment Spillovers," Journal of Development Studies, Taylor & Francis Journals, vol. 48(10), pages 1375-1396, October.
    20. Havranek, Tomas & Kokes, Ondrej, 2015. "Income elasticity of gasoline demand: A meta-analysis," Energy Economics, Elsevier, vol. 47(C), pages 77-86.

    More about this item

    Keywords

    Meta-regression; meta-analysis; publication bias; data mining; Monte Carlo simulatio;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C40 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jrp:jrpwrp:2013-040. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Markus Pasche (email available below). General contact details of provider: http://www.jenecon.de .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.