IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v126y2021i1d10.1007_s11192-020-03742-9.html
   My bibliography  Save this article

A framework for assessing the peer review duration of journals: case study in computer science

Author

Listed:
  • Besim Bilalli

    (Universitat Politècnica de Catalunya, BarcelonaTech)

  • Rana Faisal Munir

    (Eurecat Centre Tecnológic)

  • Alberto Abelló

    (Universitat Politècnica de Catalunya, BarcelonaTech)

Abstract

In various fields, scientific article publication is a measure of productivity and in many occasions it is used as a critical factor for evaluating researchers. Therefore, a lot of time is dedicated to writing articles that are then submitted for publication in journals. Nevertheless, the publication process in general and the review process in particular tend to be rather slow. This is the case for instance of computer science (CS) journals. Moreover, the process typically lacks in transparency, where information about the duration of the review process is at best provided in an aggregated manner, if made available at all. In this paper, we develop a framework as a step towards bringing more reliable data with respect to review duration. Based on this framework, we implement a tool—journal response time (JRT), that allows for automatically extracting the review process data and helps researchers to find the average response times of journals, which can be used to study the duration of CS journals’ peer review process. The information is extracted as metadata from the published articles, when available. This study reveals that the response times publicly provided by publishers differ from the actual values obtained by JRT (e.g., for ten selected journals the average duration reported by publishers deviates by more than 500% from the actual average value calculated from the data inside the articles), which we suspect could be from the fact that, when calculating the aggregated values, publishers consider the review time of rejected articles too (including quick desk-rejections that do not require reviewers).

Suggested Citation

  • Besim Bilalli & Rana Faisal Munir & Alberto Abelló, 2021. "A framework for assessing the peer review duration of journals: case study in computer science," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 545-563, January.
    • Handle: RePEc:spr:scient:v:126:y:2021:i:1:d:10.1007_s11192-020-03742-9
    DOI: 10.1007/s11192-020-03742-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03742-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-020-03742-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Oecd, 2008. "DAC Peer Review of the European Community," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 127-261.
    2. Arie Y. Lewin, 2014. "The Peer-review Process: The Good, the Bad, the Ugly, and the Extraordinary," Management and Organization Review, The International Association for Chinese Management Research, vol. 10(2), pages 167-173, July.
    3. Björk, Bo-Christer & Solomon, David, 2013. "The publishing delay in scholarly peer-reviewed journals," Journal of Informetrics, Elsevier, vol. 7(4), pages 914-923.
    4. Oecd, 2008. "DAC Peer Review of Greece," OECD Journal on Development, OECD Publishing, vol. 7(4), pages 7-93.
    5. Cinzia Daraio & Maurizio Lenzerini & Claudio Leporelli & Henk F. Moed & Paolo Naggar & Andrea Bonaccorsi & Alessandro Bartolucci, 2016. "Data integration for research and innovation policy: an Ontology-Based Data Management approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 857-871, February.
    6. Glenn Ellison, 2011. "Is Peer Review In Decline?," Economic Inquiry, Western Economic Association International, vol. 49(3), pages 635-657, July.
    7. Janine Huisman & Jeroen Smits, 2017. "Duration and quality of the peer review process: the author’s perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 633-650, October.
    8. Ivana Drvenica & Giangiacomo Bravo & Lucija Vejmelka & Aleksandar Dekanski & Olgica Nedić, 2018. "Peer Review of Reviewers: The Author’s Perspective," Publications, MDPI, vol. 7(1), pages 1-10, December.
    9. Lewin, Arie Y., 2014. "The Peer-review Process: The Good, the Bad, the Ugly, and the Extraordinary," Management and Organization Review, Cambridge University Press, vol. 10(02), pages 167-173, July.
    10. Oecd, 2008. "DAC Peer Review of the Netherlands," OECD Journal on Development, OECD Publishing, vol. 7(3), pages 115-214.
    11. David J. Solomon & Bo-Christer Björk, 2012. "Publication fees in open access publishing: Sources of funding and factors influencing choice of journal," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(1), pages 98-107, January.
    12. Maciej J. Mrowinski & Agata Fronczak & Piotr Fronczak & Olgica Nedic & Aleksandar Dekanski, 2020. "The hurdles of academic publishing from the perspective of journal editors: a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 115-133, October.
    13. Glenn Ellison, 2002. "Evolving Standards for Academic Publishing: A q-r Theory," Journal of Political Economy, University of Chicago Press, vol. 110(5), pages 994-1034, October.
    14. David J. Solomon & Bo‐Christer Björk, 2012. "Publication fees in open access publishing: Sources of funding and factors influencing choice of journal," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(1), pages 98-107, January.
    15. Oecd, 2008. "DAC Peer Review of Denmark," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 7-125.
    16. Maciej J. Mrowinski & Agata Fronczak & Piotr Fronczak & Olgica Nedic & Marcel Ausloos, 2016. "Review time in peer review: quantitative analysis and modelling of editorial workflows," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(1), pages 271-286, April.
    17. Ofer Azar, 2003. "Rejections and the Importance of First Response Times (Or: How Many Rejections Do Others Receive?)," General Economics and Teaching 0309002, University Library of Munich, Germany.
    18. Oecd, 2008. "DAC Peer Review of Canada," OECD Journal on Development, OECD Publishing, vol. 8(4), pages 263-387.
    19. Peter Kareiva & Michelle Marvier & Sabrina West & Joy Hornisher, 2002. "Slow-moving journals hinder conservation efforts," Nature, Nature, vol. 420(6911), pages 15-15, November.
    20. Oecd, 2008. "DAC Peer Review of the United States," OECD Journal on Development, OECD Publishing, vol. 7(4), pages 95-200.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zehra Taşkın & Abdülkadir Taşkın & Güleda Doğan & Emanuel Kulczycki, 2022. "Factors affecting time to publication in information science," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7499-7515, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maciej J. Mrowinski & Agata Fronczak & Piotr Fronczak & Olgica Nedic & Aleksandar Dekanski, 2020. "The hurdles of academic publishing from the perspective of journal editors: a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 115-133, October.
    2. Zhao, Zhi-Dan & Chen, Jiahao & Lu, Yichuan & Zhao, Na & Jiang, Dazhi & Wang, Bing-Hong, 2021. "Dynamic patterns of open review process," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 582(C).
    3. Janine Huisman & Jeroen Smits, 2017. "Duration and quality of the peer review process: the author’s perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 633-650, October.
    4. Lucas Rodriguez Forti & Luiz A. Solino & Judit K. Szabo, 2021. "Trade-off between urgency and reduced editorial capacity affect publication speed in ecological and medical journals during 2020," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-9, December.
    5. Lokman Tutuncu, 2023. "All-pervading insider bias alters review time in Turkish university journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3743-3791, June.
    6. ederico Bianchi & Flaminio Squazzoni, 2022. "Can transparency undermine peer review? A simulation model of scientist behavior under open peer review [Reviewing Peer Review]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 791-800.
    7. Sun, Zhuanlan & Clark Cao, C. & Ma, Chao & Li, Yiwei, 2023. "The academic status of reviewers predicts their language use," Journal of Informetrics, Elsevier, vol. 17(4).
    8. Siddarth Srinivasan & Jamie Morgenstern, 2021. "Auctions and Peer Prediction for Academic Peer Review," Papers 2109.00923, arXiv.org, revised May 2023.
    9. Katrin Hussinger & Lorenzo Palladini, 2024. "Information accessibility and knowledge creation: the impact of Google’s withdrawal from China on scientific research," Industry and Innovation, Taylor & Francis Journals, vol. 31(6), pages 753-783, July.
    10. Xie, Yundong & Wu, Qiang & Wang, Yezhu & Hou, Li & Liu, Yuanyuan, 2024. "Does the handling time of scientific papers relate to their academic impact and social attention? Evidence from Nature, Science, and PNAS," Journal of Informetrics, Elsevier, vol. 18(2).
    11. Stephen A Gallo & Afton S Carpenter & Scott R Glisson, 2013. "Teleconference versus Face-to-Face Scientific Peer Review of Grant Application: Effects on Review Outcomes," PLOS ONE, Public Library of Science, vol. 8(8), pages 1-9, August.
    12. Ana Teresa Santos & Sandro Mendonça, 2022. "Do papers (really) match journals’ “aims and scope”? A computational assessment of innovation studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7449-7470, December.
    13. Paul Sebo & Jean Pascal Fournier & Claire Ragot & Pierre-Henri Gorioux & François R. Herrmann & Hubert Maisonneuve, 2019. "Factors associated with publication speed in general medical journals: a retrospective study of bibliometric data," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1037-1058, May.
    14. José Cendejas Bueno & Cecilia Font de Villanueva, 2015. "Convergence of inflation with a common cycle: estimating and modelling Spanish historical inflation from the 16th to the 18th centuries," Empirical Economics, Springer, vol. 48(4), pages 1643-1665, June.
    15. Shan Jiang, 2021. "Understanding authors' psychological reactions to peer reviews: a text mining approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6085-6103, July.
    16. Minxian Zheng & Kuangji Zhao & Shikui Zhao & Yantong Zhang, 2020. "Effecting variables of journal’s ranking in forestry field," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 135-151, October.
    17. Justus Haucap & Johannes Muck, 2015. "What drives the relevance and reputation of economics journals? An update from a survey among economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 849-877, June.
    18. Louise Bedsworth, 2012. "California’s local health agencies and the state’s climate adaptation strategy," Climatic Change, Springer, vol. 111(1), pages 119-133, March.
    19. Stuart Lawson, 2015. "Fee Waivers for Open Access Journals," Publications, MDPI, vol. 3(3), pages 1-13, August.
    20. María Bordons & Borja González-Albo & Luz Moreno-Solano, 2023. "Improving our understanding of open access: how it relates to funding, internationality of research and scientific leadership," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4651-4676, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:126:y:2021:i:1:d:10.1007_s11192-020-03742-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.