IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v85y2010i1d10.1007_s11192-010-0233-5.html
   My bibliography  Save this article

Worsening file-drawer problem in the abstracts of natural, medical and social science databases

Author

Listed:
  • Marco Pautasso

    (Imperial College London)

Abstract

The file-drawer problem is the tendency of journals to preferentially publish studies with statistically significant results. The problem is an old one and has been documented in various fields, but to my best knowledge there has not been attention to how the issue is developing in a quantitative way through time. In the abstracts of various major scholarly databases (Science and Social Science Citation Index (1991–2008), CAB Abstracts and Medline (1970s–2008), the file drawer problem is gradually getting worse, in spite of an increase in (1) the total number of publications and (2) the proportion of publications reporting both the presence and the absence of significant differences. The trend is confirmed for particular natural science topics such as biology, energy and environment but not for papers retrieved with the keywords biodiversity, chemistry, computer, engineering, genetics, psychology and quantum (physics). A worsening file-drawer problem can be detected in various medical fields (infection, immunology, malaria, obesity, oncology and pharmacology), but not for papers indexed with strings such as AIDS/HIV, epidemiology, health and neurology. An increase in the selective publication of some results against some others is worrying because it can lead to enhanced bias in meta-analysis and hence to a distorted picture of the evidence for or against a certain hypothesis. Long-term monitoring of the file-drawer problem is needed to ensure a sustainable and reliable production of (peer-reviewed) scientific knowledge.

Suggested Citation

  • Marco Pautasso, 2010. "Worsening file-drawer problem in the abstracts of natural, medical and social science databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 193-202, October.
    • Handle: RePEc:spr:scient:v:85:y:2010:i:1:d:10.1007_s11192-010-0233-5
    DOI: 10.1007/s11192-010-0233-5
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-010-0233-5
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-010-0233-5?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Philip E Bourne & Alon Korngreen, 2006. "Ten Simple Rules for Reviewers," PLOS Computational Biology, Public Library of Science, vol. 2(9), pages 1-2, September.
    2. Colin B. Begg & Jesse A. Berlin, 1988. "Publication Bias: A Problem in Interpreting Medical Data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 151(3), pages 419-445, May.
    3. Julie Thompson Klein, 2006. "Afterword: the emergent literature on interdisciplinary and transdisciplinary research evaluation," Research Evaluation, Oxford University Press, vol. 15(1), pages 75-80, April.
    4. Peter A. Lawrence, 2003. "The politics of publication," Nature, Nature, vol. 422(6929), pages 259-261, March.
    5. Martin Reinhart, 2009. "Peer review of grant applications in biology and medicine. Reliability, fairness, and validity," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 789-809, December.
    6. Louis Mesnard, 2010. "On Hochberg et al.’s “The tragedy of the reviewer commons”," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 903-917, September.
    7. Pautasso, Marco & Pautasso, Cesare, 2010. "Peer Reviewing Interdisciplinary Papers," European Review, Cambridge University Press, vol. 18(2), pages 227-237, May.
    8. Gianmarco Paris & Giulio De Leo & Paolo Menozzi & Marino Gatto, 1998. "Region-based citation bias in science," Nature, Nature, vol. 396(6708), pages 210-210, November.
    9. Marco Pautasso & Hanno Schäfer, 2010. "Peer review delay and selectivity in ecology journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(2), pages 307-315, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Daniele Fanelli & Vincent Larivière, 2016. "Researchers’ Individual Publication Rate Has Not Increased in a Century," PLOS ONE, Public Library of Science, vol. 11(3), pages 1-12, March.
    2. Marco Pautasso, 2012. "Publication Growth in Biological Sub-Fields: Patterns, Predictability and Sustainability," Sustainability, MDPI, vol. 4(12), pages 1-14, November.
    3. Daniele Fanelli, 2013. "Positive results receive more citations, but only in some disciplines," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 701-709, February.
    4. David R Shanks & Ben R Newell & Eun Hee Lee & Divya Balakrishnan & Lisa Ekelund & Zarus Cenac & Fragkiski Kavvadia & Christopher Moore, 2013. "Priming Intelligent Behavior: An Elusive Phenomenon," PLOS ONE, Public Library of Science, vol. 8(4), pages 1-10, April.
    5. Daniele Fanelli, 2012. "Negative results are disappearing from most disciplines and countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 891-904, March.
    6. Victor A Sanchez-Azanza & Raúl López-Penadés & Lucía Buil-Legaz & Eva Aguilar-Mediavilla & Daniel Adrover-Roig, 2017. "Is bilingualism losing its advantage? A bibliometric approach," PLOS ONE, Public Library of Science, vol. 12(4), pages 1-13, April.
    7. Christine R Harris & Noriko Coburn & Doug Rohrer & Harold Pashler, 2013. "Two Failures to Replicate High-Performance-Goal Priming Effects," PLOS ONE, Public Library of Science, vol. 8(8), pages 1-1, August.
    8. Daniele Fanelli & Rodrigo Costas & Vincent Larivière, 2015. "Misconduct Policies, Academic Culture and Career Stage, Not Gender or Pressures to Publish, Affect Scientific Integrity," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-18, June.
    9. Xueying Liu & Haoran Zhu, 2023. "Linguistic positivity in soft and hard disciplines: temporal dynamics, disciplinary variation, and the relationship with research impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 3107-3127, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Vahid Garousi & João M. Fernandes, 2017. "Quantity versus impact of software engineering papers: a quantitative study," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(2), pages 963-1006, August.
    2. Stanley, T.D. & Doucouliagos, Chris & Jarrell, Stephen B., 2008. "Meta-regression analysis as the socio-economics of economics research," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 37(1), pages 276-292, February.
    3. Alexander Kalgin & Olga Kalgina & Anna Lebedeva, 2019. "Publication Metrics as a Tool for Measuring Research Productivity and Their Relation to Motivation," Voprosy obrazovaniya / Educational Studies Moscow, National Research University Higher School of Economics, issue 1, pages 44-86.
    4. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    5. Peter A Lawrence, 2009. "Real Lives and White Lies in the Funding of Scientific Research," PLOS Biology, Public Library of Science, vol. 7(9), pages 1-4, September.
    6. Chatfield, Chris, 1995. "Positive or negative?," International Journal of Forecasting, Elsevier, vol. 11(4), pages 501-502, December.
    7. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    8. Mattia Filomena & Matteo Picchio, 2023. "Retirement and health outcomes in a meta‐analytical framework," Journal of Economic Surveys, Wiley Blackwell, vol. 37(4), pages 1120-1155, September.
    9. van den Besselaar, Peter & Sandström, Ulf, 2015. "Early career grants, performance, and careers: A study on predictive validity of grant decisions," Journal of Informetrics, Elsevier, vol. 9(4), pages 826-838.
    10. Auspurg Katrin & Hinz Thomas, 2011. "What Fuels Publication Bias?: Theoretical and Empirical Analyses of Risk Factors Using the Caliper Test," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 231(5-6), pages 636-660, October.
    11. Jochen Krauss, 2007. "Journal self-citation rates in ecological sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 73(1), pages 79-89, October.
    12. Rosenberger, Randall S. & Stanley, Tom D., 2006. "Measurement, generalization, and publication: Sources of error in benefit transfers and their management," Ecological Economics, Elsevier, vol. 60(2), pages 372-378, December.
    13. Kilpatrick, Henry E., Jr., 1998. "Some useful methods for measuring the benefits of social science research," Impact assessments 5, International Food Policy Research Institute (IFPRI).
    14. Nino Fonseca & Marcelino Sánchez-Rivero, 2020. "Significance bias in the tourism-led growth literature," Tourism Economics, , vol. 26(1), pages 137-154, February.
    15. Tomas Kucera, 2020. "Are Employment Effects of Minimum Wage the Same Across the EU? A Meta-Regression Analysis," Working Papers IES 2020/2, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Jan 2020.
    16. Liang Meng & Haifeng Wang & Pengfei Han, 2020. "Getting a head start: turn-of-the-month submission effect for accepted papers in management journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2577-2595, September.
    17. Balázs Győrffy & Andrea Magda Nagy & Péter Herman & Ádám Török, 2018. "Factors influencing the scientific performance of Momentum grant holders: an evaluation of the first 117 research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 409-426, October.
    18. Jamal BOUOIYOUR & Refk SELMI & Ilhan OZTURK, 2014. "The Nexus between Electricity Consumption and Economic Growth: New Insights from Meta-Analysis," International Journal of Energy Economics and Policy, Econjournals, vol. 4(4), pages 621-635.
    19. Kuen‐Hung Tsai & Jiann‐Chyuan Wang, 2004. "R&D Productivity and the Spillover Effects of High‐tech Industry on the Traditional Manufacturing Sector: The Case of Taiwan," The World Economy, Wiley Blackwell, vol. 27(10), pages 1555-1570, November.
    20. Mora Rodríguez, Jhon James & Muro, Juan, 2015. "On the size of sheepskin effects: A meta-analysis," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 9, pages 1-18.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:85:y:2010:i:1:d:10.1007_s11192-010-0233-5. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.