IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v125y2020i1d10.1007_s11192-020-03545-y.html
   My bibliography  Save this article

Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ

Author

Listed:
  • Qianjin Zong

    (South China Normal University)

  • Yafen Xie

    (Guangdong Provincial Work Injury Rehabilitation Hospital)

  • Jiechun Liang

    (South China Normal University)

Abstract

This study aims to investigate whether open peer review can improve citation count. Articles published in PeerJ during 2013 and 2015 were chosen as the data set. Two categories of the articles were generated: articles with closed peer review history and articles with open peer review history. A propensity score matching with the radius matching method was performed using 14 confounding variables. The other five common matching methods of propensity score matching, the bias-adjusted matching estimator developed by Abadie and Imbens (Simple and bias-corrected matching estimators for average treatment effects, National Bureau of Economic Research, Cambridge, pp 1–57, 2002), and the data set excluding articles with an extremely high citation count were used to check the robustness of the results. The results of this study showed that articles with open peer review history could be expected to have significantly greater citation counts than articles with closed peer review history. Our results suggest that open peer review can improve citation count, and that the best practice for open peer review might be a compromise open peer review.

Suggested Citation

  • Qianjin Zong & Yafen Xie & Jiechun Liang, 2020. "Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 607-623, October.
    • Handle: RePEc:spr:scient:v:125:y:2020:i:1:d:10.1007_s11192-020-03545-y
    DOI: 10.1007/s11192-020-03545-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03545-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-020-03545-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Philip M. Davis & Michael J. Fromerth, 2007. "Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles?," Scientometrics, Springer;Akadémiai Kiadó, vol. 71(2), pages 203-215, May.
    2. Alberto Abadie & David Drukker & Jane Leber Herr & Guido W. Imbens, 2004. "Implementing matching estimators for average treatment effects in Stata," Stata Journal, StataCorp LP, vol. 4(3), pages 290-311, September.
    3. Feng Guo & Chao Ma & Qingling Shi & Qingqing Zong, 2018. "Succinct effect or informative effect: the relationship between title length and the number of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1531-1539, September.
    4. Lutz Bornmann & Markus Wolf & Hans-Dieter Daniel, 2012. "Closed versus open reviewing of journal manuscripts: how far do comments differ in language use?," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 843-856, June.
    5. Matthias Gnewuch & Klaus Wohlrabe, 2017. "Title characteristics and citations in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1573-1578, March.
    6. Alberto Falk Delgado & Gregory Garretson & Anna Falk Delgado, 2019. "The language of peer review reports on articles published in the BMJ, 2014–2017: an observational study," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(3), pages 1225-1235, September.
    7. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
    8. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    9. Jeffrey T Leek & Margaret A Taub & Fernando J Pineda, 2011. "Cooperation between Referees and Authors Increases Peer Review Accuracy," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-11, November.
    10. Uddin, Shahadat & Khan, Arif, 2016. "The impact of author-selected keywords on citation counts," Journal of Informetrics, Elsevier, vol. 10(4), pages 1166-1177.
    11. Afshin Sadeghi & Sarven Capadisli & Johannes Wilm & Christoph Lange & Philipp Mayr, 2019. "Opening and Reusing Transparent Peer Reviews with Automatic Article Annotation," Publications, MDPI, vol. 7(1), pages 1-12, February.
    12. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    13. Shi V. Liu, 2007. "Why are people reluctant to join in open review?," Nature, Nature, vol. 447(7148), pages 1052-1052, June.
    14. Alberto Abadie & Guido W. Imbens, 2002. "Simple and Bias-Corrected Matching Estimators for Average Treatment Effects," NBER Technical Working Papers 0283, National Bureau of Economic Research, Inc.
    15. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    16. Fereshteh Didegah & Timothy D. Bowman & Kim Holmberg, 2018. "On the differences between citations and altmetrics: An investigation of factors driving altmetrics versus citations for finnish articles," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(6), pages 832-843, June.
    17. Lutz Bornmann & Werner Marx & Hermann Schier & Andreas Thor & Hans-Dieter Daniel, 2010. "From black box to white box at open access journals: predictive validity of manuscript reviewing and editorial decisions at Atmospheric Chemistry and Physics," Research Evaluation, Oxford University Press, vol. 19(2), pages 105-118, June.
    18. Michael S. Patterson & Simon Harris, 2009. "The relationship between reviewers’ quality-scores and number of citations for papers published in the journal Physics in Medicine and Biology from 2003–2005," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(2), pages 343-349, August.
    19. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    20. Alberto Abadie & Guido W. Imbens, 2006. "Large Sample Properties of Matching Estimators for Average Treatment Effects," Econometrica, Econometric Society, vol. 74(1), pages 235-267, January.
    21. Syavash Nobarany & Kellogg S. Booth, 2015. "Use of politeness strategies in signed open peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(5), pages 1048-1064, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Qianjin Zong & Zhihong Huang & Jiaru Huang, 2023. "Do open science badges work? Estimating the effects of open science badges on an article’s social media attention and research impacts," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3627-3648, June.
    2. Eirini Delikoura & Dimitrios Kouis, 2021. "Open Research Data and Open Peer Review: Perceptions of a Medical and Health Sciences Community in Greece," Publications, MDPI, vol. 9(2), pages 1-19, March.
    3. Fan, Lingxu & Guo, Lei & Wang, Xinhua & Xu, Liancheng & Liu, Fangai, 2022. "Does the author’s collaboration mode lead to papers’ different citation impacts? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 16(4).
    4. Cheng, Xi & Wang, Haoran & Tang, Li & Jiang, Weiyan & Zhou, Maotian & Wang, Guoyan, 2024. "Open peer review correlates with altmetrics but not with citations: Evidence from Nature Communications and PLoS One," Journal of Informetrics, Elsevier, vol. 18(3).
    5. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).
    6. Weixi Xie & Pengfei Jia & Guangyao Zhang & Xianwen Wang, 2024. "Are reviewer scores consistent with citations?," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4721-4740, August.
    7. Borenstein, Denis & Perlin, Marcelo S. & Imasato, Takeyoshi, 2022. "The Academic Inbreeding Controversy: Analysis and Evidence from Brazil," Journal of Informetrics, Elsevier, vol. 16(2).
    8. Chunli Wei & Jingyi Zhao & Jue Ni & Jiang Li, 2023. "What does open peer review bring to scientific articles? Evidence from PLoS journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 2763-2776, May.
    9. Hou, Li & Wu, Qiang & Xie, Yundong, 2024. "Does open identity of peer reviewers positively relate to citations?," Journal of Informetrics, Elsevier, vol. 18(1).
    10. Junping Qiu & Yunlong Yu & Shiji Chen & Teng Zhao & Shanshan Wang, 2024. "Effect of Scientific Collaboration on Interdisciplinarity in Climate Change From a Scientometric Perspective," SAGE Open, , vol. 14(2), pages 21582440241, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dietmar Wolfram & Peiling Wang & Adam Hembree & Hyoungjoo Park, 2020. "Open peer review: promoting transparency in open science," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1033-1051, November.
    2. Rodríguez Sánchez, Isabel & Makkonen, Teemu & Williams, Allan M., 2019. "Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers," Annals of Tourism Research, Elsevier, vol. 77(C), pages 1-11.
    3. Yuetong Chen & Hao Wang & Baolong Zhang & Wei Zhang, 2022. "A method of measuring the article discriminative capacity and its distribution," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3317-3341, June.
    4. Eirini Delikoura & Dimitrios Kouis, 2021. "Open Research Data and Open Peer Review: Perceptions of a Medical and Health Sciences Community in Greece," Publications, MDPI, vol. 9(2), pages 1-19, March.
    5. Kong, Ling & Wang, Dongbo, 2020. "Comparison of citations and attention of cover and non-cover papers," Journal of Informetrics, Elsevier, vol. 14(4).
    6. Ostapchuk, Igor & Gagalyuk, Taras & Curtiss, Jarmila, 2021. "Post-acquisition integration and growth of farms: the case of Ukrainian agroholdings," International Food and Agribusiness Management Review, International Food and Agribusiness Management Association, vol. 24(4), April.
    7. Shen, Chung-Hua & Wu, Meng-Wen & Chen, Ting-Hsuan & Fang, Hao, 2016. "To engage or not to engage in corporate social responsibility: Empirical evidence from global banking sector," Economic Modelling, Elsevier, vol. 55(C), pages 207-225.
    8. Davis, K. & Nkonya, E. & Kato, E. & Mekonnen, D.A. & Odendo, M. & Miiro, R. & Nkuba, J., 2012. "Impact of Farmer Field Schools on Agricultural Productivity and Poverty in East Africa," World Development, Elsevier, vol. 40(2), pages 402-413.
    9. Qianjin Zong & Yafen Xie & Rongchan Tuo & Jingshi Huang & Yang Yang, 2019. "The impact of video abstract on citation counts: evidence from a retrospective cohort study of New Journal of Physics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1715-1727, June.
    10. Petracco, Carly K. & Pender, John L., 2009. "Evaluating the Impact of Land Tenure and Titling on Access to Credit in Uganda," 2009 Conference, August 16-22, 2009, Beijing, China 51899, International Association of Agricultural Economists.
    11. Gueorgui Kambourov & Iourii Manovskii & Miana Plesca, 2020. "Occupational mobility and the returns to training," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 53(1), pages 174-211, February.
    12. Steven Lehrer & Gregory Kordas, 2013. "Matching using semiparametric propensity scores," Empirical Economics, Springer, vol. 44(1), pages 13-45, February.
    13. Cheng, Xi & Wang, Haoran & Tang, Li & Jiang, Weiyan & Zhou, Maotian & Wang, Guoyan, 2024. "Open peer review correlates with altmetrics but not with citations: Evidence from Nature Communications and PLoS One," Journal of Informetrics, Elsevier, vol. 18(3).
    14. Hird, Mackenzie D. & Pfotenhauer, Sebastian M., 2017. "How complex international partnerships shape domestic research clusters: Difference-in-difference network formation and research re-orientation in the MIT Portugal Program," Research Policy, Elsevier, vol. 46(3), pages 557-572.
    15. Yu Bai & Linxiu Zhang & Chengfang Liu & Yaojiang Shi & Di Mo & Scott Rozelle, 2018. "Effect of Parental Migration on the Academic Performance of Left Behind Children in North Western China," Journal of Development Studies, Taylor & Francis Journals, vol. 54(7), pages 1154-1170, July.
    16. Emilija Stojmenova Duh & Andrej Duh & Uroš Droftina & Tim Kos & Urban Duh & Tanja Simonič Korošak & Dean Korošak, 2019. "Publish-and-Flourish: Using Blockchain Platform to Enable Cooperative Scholarly Communication," Publications, MDPI, vol. 7(2), pages 1-15, May.
    17. Palmer, Michael G., 2014. "Inequalities in Universal Health Coverage: Evidence from Vietnam," World Development, Elsevier, vol. 64(C), pages 384-394.
    18. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
    19. Alberto Abadie & Guido W. Imbens, 2008. "On the Failure of the Bootstrap for Matching Estimators," Econometrica, Econometric Society, vol. 76(6), pages 1537-1557, November.
    20. Peng, Wen & Yue, Mingliang & Sun, Mingyue & Ma, Tingcan, 2024. "Revision and academic impact: A case study of bioRxiv preprint papers," Journal of Informetrics, Elsevier, vol. 18(1).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:125:y:2020:i:1:d:10.1007_s11192-020-03545-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.