IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v37y2008i1p3-30.html
   My bibliography  Save this article

Publication Bias in Empirical Sociological Research

Author

Listed:
  • Alan S. Gerber

    (Yale University, New Haven, Connecticut)

  • Neil Malhotra

    (Stanford University, Stanford, California, neilm@stanford.edu)

Abstract

Despite great attention to the quality of research methods in individual studies, if publication decisions of journals are a function of the statistical significance of research findings, the published literature as a whole may not produce accurate measures of true effects. This article examines the two most prominent sociology journals (the American Sociological Review and the American Journal of Sociology ) and another important though less influential journal ( The Sociological Quarterly ) for evidence of publication bias. The effect of the .05 significance level on the pattern of published findings is examined using a ``caliper'' test, and the hypothesis of no publication bias can be rejected at approximately the 1 in 10 million level. Findings suggest that some of the results reported in leading sociology journals may be misleading and inaccurate due to publication bias. Some reasons for publication bias and proposed reforms to reduce its impact on research are also discussed.

Suggested Citation

  • Alan S. Gerber & Neil Malhotra, 2008. "Publication Bias in Empirical Sociological Research," Sociological Methods & Research, , vol. 37(1), pages 3-30, August.
    • Handle: RePEc:sae:somere:v:37:y:2008:i:1:p:3-30
    DOI: 10.1177/0049124108318973
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124108318973
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0049124108318973?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Devah Pager, 2003. "The mark of a criminal record," Natural Field Experiments 00319, The Field Experiments Website.
    2. T. D. Stanley & Stephen B. Jarrell, 2005. "Meta‐Regression Analysis: A Quantitative Method of Literature Surveys," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 299-308, July.
    3. Sigelman, Lee, 1999. "Publication Bias Reconsidered," Political Analysis, Cambridge University Press, vol. 8(2), pages 201-210, December.
    4. Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-243, May.
    5. De Long, J Bradford & Lang, Kevin, 1992. "Are All Economic Hypotheses False?," Journal of Political Economy, University of Chicago Press, vol. 100(6), pages 1257-1272, December.
    6. Colin B. Begg & Jesse A. Berlin, 1988. "Publication Bias: A Problem in Interpreting Medical Data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 151(3), pages 419-445, May.
    7. J. Copas, 1999. "What works?: selectivity models and meta‐analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 162(1), pages 95-109.
    8. Orley Ashenfelter & Colm Harmon & Hessel Oosterbeek, 1999. "A Review of Estimates of the Schooling/Earnings Relationship, with Tests for Publication Bias," Working Papers 804, Princeton University, Department of Economics, Industrial Relations Section..
    9. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    10. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    11. repec:fth:prinin:425 is not listed on IDEAS
    12. Chris Doucouliagos, 2005. "Publication Bias in the Economic Freedom and Economic Growth Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 367-387, July.
    13. Chris (hristos) Doucouliagos & Patrice Laroche & T.D. Stanley, 2005. "Publication Bias in Union-Productivity Research? [¿ Publicación tendenciosa en la investigación sobre la productividad sindical ?]," Post-Print hal-02138187, HAL.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stanley, T. D. & Doucouliagos, Hristos, 2013. "Better than random: weighted least squares meta-regression analysis," Working Papers eco_2013_2, Deakin University, Department of Economics.
    2. Auspurg Katrin & Hinz Thomas, 2011. "What Fuels Publication Bias?: Theoretical and Empirical Analyses of Risk Factors Using the Caliper Test," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 231(5-6), pages 636-660, October.
    3. Weiß Bernd & Wagner Michael, 2011. "The Identification and Prevention of Publication Bias in the Social Sciences and Economics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 231(5-6), pages 661-684, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    2. Hristos Doucouliagos & T. D. Stanley, 2009. "Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 47(2), pages 406-428, June.
    3. Stanley, T.D. & Doucouliagos, Chris & Jarrell, Stephen B., 2008. "Meta-regression analysis as the socio-economics of economics research," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 37(1), pages 276-292, February.
    4. Stanley, T. D. & Doucouliagos, Hristos, 2011. "Meta-regression approximations to reduce publication selection bias," Working Papers eco_2011_4, Deakin University, Department of Economics.
    5. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    6. Neves, Pedro Cunha & Afonso, Óscar & Silva, Sandra Tavares, 2016. "A Meta-Analytic Reassessment of the Effects of Inequality on Growth," World Development, Elsevier, vol. 78(C), pages 386-400.
    7. Fidrmuc, Jarko & Lind, Ronja, 2020. "Macroeconomic impact of Basel III: Evidence from a meta-analysis," Journal of Banking & Finance, Elsevier, vol. 112(C).
    8. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    9. Mattia Filomena & Matteo Picchio, 2023. "Retirement and health outcomes in a meta‐analytical framework," Journal of Economic Surveys, Wiley Blackwell, vol. 37(4), pages 1120-1155, September.
    10. Rosenberger, Randall S. & Stanley, Tom D., 2006. "Measurement, generalization, and publication: Sources of error in benefit transfers and their management," Ecological Economics, Elsevier, vol. 60(2), pages 372-378, December.
    11. Tomas Havranek, 2013. "Publication Bias in Measuring Intertemporal Substitution," Working Papers IES 2013/15, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Oct 2013.
    12. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    13. Havranek, Tomas & Irsova, Zuzana & Janda, Karel, 2012. "Demand for gasoline is more price-inelastic than commonly thought," Energy Economics, Elsevier, vol. 34(1), pages 201-207.
    14. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    15. Maria Cipollina & Luca Salvatici, 2010. "Reciprocal Trade Agreements in Gravity Models: A Meta‐Analysis," Review of International Economics, Wiley Blackwell, vol. 18(1), pages 63-80, February.
    16. Havranek, Tomas & Horvath, Roman & Irsova, Zuzana & Rusnak, Marek, 2015. "Cross-country heterogeneity in intertemporal substitution," Journal of International Economics, Elsevier, vol. 96(1), pages 100-118.
    17. Snyder, Christopher & Zhuo, Ran, 2018. "Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias," MetaArXiv 8vdrh, Center for Open Science.
    18. Liu, Boying & Richard Shumway, C., 2016. "Substitution elasticities between GHG-polluting and nonpolluting inputs in agricultural production: A meta-regression," Energy Economics, Elsevier, vol. 54(C), pages 123-132.
    19. Costa-Font, Joan & McGuire, Alistair & Stanley, Tom, 2013. "Publication selection in health policy research: The winner's curse hypothesis," Health Policy, Elsevier, vol. 109(1), pages 78-87.
    20. Crespo Cuaresma, Jesús & Fidrmuc, Jarko & Hake, Mariya, 2014. "Demand and supply drivers of foreign currency loans in CEECs: A meta-analysis," Economic Systems, Elsevier, vol. 38(1), pages 26-42.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:37:y:2008:i:1:p:3-30. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.