IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v106y2016i2d10.1007_s11192-015-1800-6.html
   My bibliography  Save this article

Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data

Author

Listed:
  • Michail Kovanis

    (INSERM U1153
    Université Paris Descartes – Sorbonne Paris Cité)

  • Raphaël Porcher

    (INSERM U1153
    Université Paris Descartes – Sorbonne Paris Cité
    Assistance Publique-Hôpitaux de Paris)

  • Philippe Ravaud

    (INSERM U1153
    Université Paris Descartes – Sorbonne Paris Cité
    Assistance Publique-Hôpitaux de Paris
    Cochrane France)

  • Ludovic Trinquart

    (INSERM U1153
    Cochrane France)

Abstract

Scientific peer-review and publication systems incur a huge burden in terms of costs and time. Innovative alternatives have been proposed to improve the systems, but assessing their impact in experimental studies is not feasible at a systemic level. We developed an agent-based model by adopting a unified view of peer review and publication systems and calibrating it with empirical journal data in the biomedical and life sciences. We modeled researchers, research manuscripts and scientific journals as agents. Researchers were characterized by their scientific level and resources, manuscripts by their scientific value, and journals by their reputation and acceptance or rejection thresholds. These state variables were used in submodels for various processes such as production of articles, submissions to target journals, in-house and external peer review, and resubmissions. We collected data for a sample of biomedical and life sciences journals regarding acceptance rates, resubmission patterns and total number of published articles. We adjusted submodel parameters so that the agent-based model outputs fit these empirical data. We simulated 105 journals, 25,000 researchers and 410,000 manuscripts over 10 years. A mean of 33,600 articles were published per year; 19 % of submitted manuscripts remained unpublished. The mean acceptance rate was 21 % after external peer review and rejection rate 32 % after in-house review; 15 % publications resulted from the first submission, 47 % the second submission and 20 % the third submission. All decisions in the model were mainly driven by the scientific value, whereas journal targeting and persistence in resubmission defined whether a manuscript would be published or abandoned after one or many rejections. This agent-based model may help in better understanding the determinants of the scientific publication and peer-review systems. It may also help in assessing and identifying the most promising alternative systems of peer review.

Suggested Citation

  • Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 695-715, February.
    • Handle: RePEc:spr:scient:v:106:y:2016:i:2:d:10.1007_s11192-015-1800-6
    DOI: 10.1007/s11192-015-1800-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-015-1800-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-015-1800-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    2. In-Uck Park & Mike W. Peacey & Marcus R. Munafò, 2014. "Modelling the effects of subjective and objective decision making in scientific peer review," Nature, Nature, vol. 506(7486), pages 93-96, February.
    3. Mario Paolucci & Francisco Grimaldo, 2014. "Mechanism change in a simulation of peer review: from junk support to elitism," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(3), pages 663-688, June.
    4. S. Thurner & R. Hanel, 2011. "Peer-review in a world with rational scientists: Toward selection of the average," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 84(4), pages 707-711, December.
    5. Trisha Gura, 2002. "Peer review, unmasked," Nature, Nature, vol. 416(6878), pages 258-260, March.
    6. Flaminio Squazzoni & Claudio Gandelli, 2013. "Opening the Black-Box of Peer Review: An Agent-Based Model of Scientist Behaviour," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 16(2), pages 1-3.
    7. Flaminio Squazzoni, 2010. "The impact of agent-based models in the social sciences after 15 years of incursions," History of Economic Ideas, Fabrizio Serra Editore, Pisa - Roma, vol. 18(2), pages 197-234.
    8. Martijn Arns, 2014. "Open access is tiring out peer reviewers," Nature, Nature, vol. 515(7528), pages 467-467, November.
    9. J. Doyne Farmer & Duncan Foley, 2009. "The economy needs agent-based modelling," Nature, Nature, vol. 460(7256), pages 685-686, August.
    10. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    11. Jagpreet Chhatwal & Tianhua He, 2015. "Economic Evaluations with Agent-Based Modelling: An Introduction," PharmacoEconomics, Springer, vol. 33(5), pages 423-433, May.
    12. Day, Theodore Eugene, 2015. "The big consequences of small biases: A simulation of peer review," Research Policy, Elsevier, vol. 44(6), pages 1266-1270.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    2. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2021. "The interplay between the reviewer’s incentives and the journal’s quality standard," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3041-3061, April.
    3. Simone Righi & Károly Takács, 2017. "The miracle of peer review and development in science: an agent-based model," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 587-607, October.
    4. Michail Kovanis & Ludovic Trinquart & Philippe Ravaud & Raphaël Porcher, 2017. "Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 651-671, October.
    5. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2020. "The author–reviewer game," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2409-2431, September.
    6. Ausloos, Marcel & Nedic, Olgica & Dekanski, Aleksandar & Mrowinski, Maciej J. & Fronczak, Piotr & Fronczak, Agata, 2017. "Day of the week effect in paper submission/acceptance/rejection to/in/by peer review journals. II. An ARCH econometric-like modeling," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 468(C), pages 462-474.
    7. Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-14, November.
    8. Francisco Grimaldo & Mario Paolucci & Jordi Sabater-Mir, 2018. "Reputation or peer review? The role of outliers," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1421-1438, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michail Kovanis & Ludovic Trinquart & Philippe Ravaud & Raphaël Porcher, 2017. "Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 651-671, October.
    2. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2021. "The interplay between the reviewer’s incentives and the journal’s quality standard," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3041-3061, April.
    3. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    4. Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-14, November.
    5. Rodríguez Sánchez, Isabel & Makkonen, Teemu & Williams, Allan M., 2019. "Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers," Annals of Tourism Research, Elsevier, vol. 77(C), pages 1-11.
    6. Pawel Sobkowicz, 2015. "Innovation Suppression and Clique Evolution in Peer-Review-Based, Competitive Research Funding Systems: An Agent-Based Model," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 18(2), pages 1-13.
    7. Federico Bianchi & Francisco Grimaldo & Giangiacomo Bravo & Flaminio Squazzoni, 2018. "The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1401-1420, September.
    8. Maciej J. Mrowinski & Agata Fronczak & Piotr Fronczak & Olgica Nedic & Marcel Ausloos, 2016. "Review time in peer review: quantitative analysis and modelling of editorial workflows," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(1), pages 271-286, April.
    9. Alessandro Checco & Lorenzo Bracciale & Pierpaolo Loreti & Stephen Pinfield & Giuseppe Bianchi, 2021. "AI-assisted peer review," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-11, December.
    10. Kuklin, Alexander A. (Куклин, Александр) & Balyakina, Evgeniya A. (Балякина, Евгения), 2017. "Active policy as a key to success for an International Economic Periodical [Активная Политика — Залог Успеха Международного Экономического Журнала]," Ekonomicheskaya Politika / Economic Policy, Russian Presidential Academy of National Economy and Public Administration, vol. 6, pages 160-177, December.
    11. Feliciani, Thomas & Morreau, Michael & Luo, Junwen & Lucas, Pablo & Shankar, Kalpana, 2022. "Designing grant-review panels for better funding decisions: Lessons from an empirically calibrated simulation model," Research Policy, Elsevier, vol. 51(4).
    12. Vivian M Nguyen & Neal R Haddaway & Lee F G Gutowsky & Alexander D M Wilson & Austin J Gallagher & Michael R Donaldson & Neil Hammerschlag & Steven J Cooke, 2015. "How Long Is Too Long in Contemporary Peer Review? Perspectives from Authors Publishing in Conservation Biology Journals," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-20, August.
    13. Dietmar Wolfram & Peiling Wang & Adam Hembree & Hyoungjoo Park, 2020. "Open peer review: promoting transparency in open science," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1033-1051, November.
    14. Bianchi, Federico & Grimaldo, Francisco & Squazzoni, Flaminio, 2019. "The F3-index. Valuing reviewers for scholarly journals," Journal of Informetrics, Elsevier, vol. 13(1), pages 78-86.
    15. Yuetong Chen & Hao Wang & Baolong Zhang & Wei Zhang, 2022. "A method of measuring the article discriminative capacity and its distribution," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3317-3341, June.
    16. Paul Sebo & Jean Pascal Fournier & Claire Ragot & Pierre-Henri Gorioux & François R. Herrmann & Hubert Maisonneuve, 2019. "Factors associated with publication speed in general medical journals: a retrospective study of bibliometric data," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1037-1058, May.
    17. Qianjin Zong & Yafen Xie & Jiechun Liang, 2020. "Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 607-623, October.
    18. Maciej J Mrowinski & Piotr Fronczak & Agata Fronczak & Marcel Ausloos & Olgica Nedic, 2017. "Artificial intelligence in peer review: How can evolutionary computation support journal editors?," PLOS ONE, Public Library of Science, vol. 12(9), pages 1-11, September.
    19. Francisco Grimaldo & Mario Paolucci & Jordi Sabater-Mir, 2018. "Reputation or peer review? The role of outliers," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1421-1438, September.
    20. Eirini Delikoura & Dimitrios Kouis, 2021. "Open Research Data and Open Peer Review: Perceptions of a Medical and Health Sciences Community in Greece," Publications, MDPI, vol. 9(2), pages 1-19, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:106:y:2016:i:2:d:10.1007_s11192-015-1800-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.