IDEAS home Printed from https://ideas.repec.org/a/eee/ininma/v46y2019icp286-293.html
   My bibliography  Save this article

An overview of assessing the quality of peer review reports of scientific articles

Author

Listed:
  • Sizo, Amanda
  • Lino, Adriano
  • Reis, Luis Paulo
  • Rocha, Álvaro

Abstract

Assuring the quality control of publications in the scientific literature is one of the main challenges of the peer review process. Consequently, there has been an increasing demand for computing solutions that will help to maintain the quality of this process. Recently, the use of Artificial Intelligence techniques has been highlighted, applied in the detection of plagiarism, bias, among other functions. The assessment of the reviewer’s review has also been considered as important in the process, but, little is known about it, for instance, which techniques have been applied in this assessment or which criteria have been assessed. Therefore, this systematic literature review aims to find evidence regarding the computational approaches that have been used to evaluate reviewers' reports. In order to achieve this, five online databases were selected, from which 72 articles were identified that met the inclusion criteria of this review, all of which have been published since 2000. The result returned 10 relevant studies meeting the evaluation requirements of scientific article reviews. The review revealed that mechanisms to rank review reports according to a score, as well as the word analysis, are the most common tools, and that there is no consensus on quality criteria. The systematic literature review has shown that reviewers’ report assessment is a valid tool for maintaining quality throughout the process. However, it still needs to be further developed if it is to be used as a resource which surpass a single conference or journal, making the peer review process more rigorous and less based on random choice.

Suggested Citation

  • Sizo, Amanda & Lino, Adriano & Reis, Luis Paulo & Rocha, Álvaro, 2019. "An overview of assessing the quality of peer review reports of scientific articles," International Journal of Information Management, Elsevier, vol. 46(C), pages 286-293.
    • Handle: RePEc:eee:ininma:v:46:y:2019:i:c:p:286-293
    DOI: 10.1016/j.ijinfomgt.2018.07.002
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0268401218304857
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijinfomgt.2018.07.002?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Lutz Bornmann & Markus Wolf & Hans-Dieter Daniel, 2012. "Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 843-856, June.
    2. Zahedi, Mansooreh & Shahin, Mojtaba & Ali Babar, Muhammad, 2016. "A systematic review of knowledge sharing challenges and practices in global software development," International Journal of Information Management, Elsevier, vol. 36(6), pages 995-1019.
    3. Balaid, Ali & Abd Rozan, Mohd Zaidi & Hikmi, Syed Norris & Memon, Jamshed, 2016. "Knowledge maps: A systematic literature review and directions for future research," International Journal of Information Management, Elsevier, vol. 36(3), pages 451-475.
    4. Rekik, Rim & Kallel, Ilhem & Casillas, Jorge & Alimi, Adel M., 2018. "Assessing web sites quality: A systematic literature review by text and association rules mining," International Journal of Information Management, Elsevier, vol. 38(1), pages 201-216.
    5. Busalim, Abdelsalam H. & Hussin, Ab Razak Che, 2016. "Understanding social commerce: A systematic literature review and directions for further research," International Journal of Information Management, Elsevier, vol. 36(6), pages 1075-1088.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. EL Idrissi, Touria & Idri, Ali & Bakkoury, Zohra, 2019. "Systematic map and review of predictive techniques in diabetes self-management," International Journal of Information Management, Elsevier, vol. 46(C), pages 263-277.
    2. Agarwal, Shweta & Kumar, Shailendra & Goel, Utkarsh, 2019. "Stock market response to information diffusion through internet sources: A literature review," International Journal of Information Management, Elsevier, vol. 45(C), pages 118-131.
    3. Al-Emran, Mostafa & Mezhuyev, Vitaliy & Kamaludin, Adzhar & Shaalan, Khaled, 2018. "The impact of knowledge management processes on information systems: A systematic review," International Journal of Information Management, Elsevier, vol. 43(C), pages 173-187.
    4. Hren, Darko & Pina, David G. & Norman, Christopher R. & Marušić, Ana, 2022. "What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports," Journal of Informetrics, Elsevier, vol. 16(2).
    5. Amiri, Babak & Karimianghadim, Ramin, 2024. "A novel text clustering model based on topic modelling and social network analysis," Chaos, Solitons & Fractals, Elsevier, vol. 181(C).
    6. Mahmud, Hasan & Islam, A.K.M. Najmul & Ahmed, Syed Ishtiaque & Smolander, Kari, 2022. "What influences algorithmic decision-making? A systematic literature review on algorithm aversion," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    7. Gupta, Manjul & George, Joey F. & Xia, Weidong, 2019. "Relationships between IT department culture and agile software development practices: An empirical investigation," International Journal of Information Management, Elsevier, vol. 44(C), pages 13-24.
    8. Agag, Gomaa & Eid, Riyad & Chaib Lababdi, Houyem & Abdelwahab, Mohamed & Aboul-Dahab, Sameh & Abdo, Said Shabban, 2024. "Understanding the impact of national culture differences on customers’ online social shopping behaviours," Journal of Retailing and Consumer Services, Elsevier, vol. 79(C).
    9. Soltanian, Salman & Kalogirou, Soteris A. & Ranjbari, Meisam & Amiri, Hamid & Mahian, Omid & Khoshnevisan, Benyamin & Jafary, Tahereh & Nizami, Abdul-Sattar & Gupta, Vijai Kumar & Aghaei, Siavash & Pe, 2022. "Exergetic sustainability analysis of municipal solid waste treatment systems: A systematic critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 156(C).
    10. Jinjie Li & Jiayin Qi & Lianren Wu & Nan Shi & Xu Li & Yuxin Zhang & Yinyin Zheng, 2021. "The Continued Use of Social Commerce Platforms and Psychological Anxiety—The Roles of Influencers, Informational Incentives and FoMO," IJERPH, MDPI, vol. 18(22), pages 1-19, November.
    11. Yogesh K Dwivedi & Elvira Ismagilova & Prianka Sarker & Anand Jeyaraj & Yassine Jadil & Laurie Hughes, 2023. "A Meta-Analytic Structural Equation Model for Understanding Social Commerce Adoption," Information Systems Frontiers, Springer, vol. 25(4), pages 1421-1437, August.
    12. Dietmar Wolfram & Peiling Wang & Adam Hembree & Hyoungjoo Park, 2020. "Open peer review: promoting transparency in open science," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1033-1051, November.
    13. Isaac Kofi Mensah & Guohua Zeng & Chuanyong Luo, 2023. "Determinants of Social Commerce Purchase and Recommendation Intentions Within the Context of Swift Guanxi Among Chinese College Students," SAGE Open, , vol. 13(2), pages 21582440231, June.
    14. Cahyono, Edi Dwi, 2023. "Instagram adoption for local food transactions: A research framework," Technological Forecasting and Social Change, Elsevier, vol. 187(C).
    15. Zhengyang Liu, 2023. "A Comprehensive Review of Affordance Theory in Enhancing Digital Health Solutions," Technium, Technium Science, vol. 19(1), pages 59-72.
    16. Vazquez, Erik Ernesto & Patel, Chirag & Alvidrez, Salvador & Siliceo, Lorena, 2023. "Images, reviews, and purchase intention on social commerce: The role of mental imagery vividness, cognitive and affective social presence," Journal of Retailing and Consumer Services, Elsevier, vol. 74(C).
    17. KYDROS Dimitrios & FILENTA Pagona, 2022. "Literature Review of Economic and Regional Development through Quantitative Methods and Social Network Analysis," European Journal of Interdisciplinary Studies, Bucharest Economic Academy, issue 01, March.
    18. Sahar Vahdati & Said Fathalla & Christoph Lange & Andreas Behrend & Aysegul Say & Zeynep Say & Sören Auer, 2021. "A comprehensive quality assessment framework for scientific events," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 641-682, January.
    19. Qianjin Zong & Yafen Xie & Jiechun Liang, 2020. "Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 607-623, October.
    20. Lundmark, Erik & Milanov, Hana & Seigner, Benedikt David Christian, 2022. "Can it be measured? A quantitative assessment of critiques of the entrepreneurship literature," Journal of Business Venturing Insights, Elsevier, vol. 17(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ininma:v:46:y:2019:i:c:p:286-293. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/international-journal-of-information-management .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.