IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v116y2018i3d10.1007_s11192-018-2826-3.html
   My bibliography  Save this article

Reputation or peer review? The role of outliers

Author

Listed:
  • Francisco Grimaldo

    (Universitat de València)

  • Mario Paolucci

    (Italian National Research Council)

  • Jordi Sabater-Mir

    (Spanish National Research Council)

Abstract

We present an agent-based model of paper publication and consumption that allows to study the effect of two different evaluation mechanisms, peer review and reputation, on the quality of the manuscripts accessed by a scientific community. The model was empirically calibrated on two data sets, mono- and multi-disciplinary. Our results point out that disciplinary settings differ in the rapidity with which they deal with extreme events—papers that have an extremely high quality, that we call outliers. In the mono-disciplinary case, reputation is better than traditional peer review to optimize the quality of papers read by researchers. In the multi-disciplinary case, if the quality landscape is relatively flat, a reputation system also performs better. In the presence of outliers, peer review is more effective. Our simulation suggests that a reputation system could perform better than peer review as a scientific information filter for quality except when research is multi-disciplinary and in a field where outliers exist.

Suggested Citation

  • Francisco Grimaldo & Mario Paolucci & Jordi Sabater-Mir, 2018. "Reputation or peer review? The role of outliers," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1421-1438, September.
    • Handle: RePEc:spr:scient:v:116:y:2018:i:3:d:10.1007_s11192-018-2826-3
    DOI: 10.1007/s11192-018-2826-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-018-2826-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-018-2826-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mario Paolucci & Francisco Grimaldo, 2014. "Mechanism change in a simulation of peer review: from junk support to elitism," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(3), pages 663-688, June.
    2. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    3. Flaminio Squazzoni & Claudio Gandelli, 2013. "Opening the Black-Box of Peer Review: An Agent-Based Model of Scientist Behaviour," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 16(2), pages 1-3.
    4. Parolo, Pietro Della Briotta & Pan, Raj Kumar & Ghosh, Rumi & Huberman, Bernardo A. & Kaski, Kimmo & Fortunato, Santo, 2015. "Attention decay in science," Journal of Informetrics, Elsevier, vol. 9(4), pages 734-745.
    5. N. Gilbert, 1997. "A Simulation of the Structure of Academic Science," Sociological Research Online, , vol. 2(2), pages 91-105, June.
    6. Simone Righi & Károly Takács, 2017. "The miracle of peer review and development in science: an agent-based model," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 587-607, October.
    7. Francisco Grimaldo & Mario Paolucci, 2013. "A Simulation Of Disagreement For Control Of Rational Cheating In Peer Review," Advances in Complex Systems (ACS), World Scientific Publishing Co. Pte. Ltd., vol. 16(07), pages 1-24.
    8. Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 695-715, February.
    9. Bruce Edmonds & Nigel Gilbert & Petra Ahrweiler & Andrea Scharnhorst, 2011. "Simulating the Social Processes of Science," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 14(4), pages 1-14.
    10. Xin Gu & Karen Blackmore & David Cornforth & Keith Nesbitt, 2015. "Modelling Academics as Agents: An Implementation of an Agent-Based Strategic Publication Model," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 18(2), pages 1-10.
    11. Nicole J. Saam & L. Reiter, 1999. "Lotka's law reconsidered: The evolution of publication and citation distributions in scientific fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 44(2), pages 135-155, February.
    12. Matthias Meyer, 2011. "Bibliometrics, Stylized Facts and the Way Ahead: How to Build Good Social Simulation Models of Science?," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 14(4), pages 1-4.
    13. Michail Kovanis & Ludovic Trinquart & Philippe Ravaud & Raphaël Porcher, 2017. "Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 651-671, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Thomas Feliciani & Junwen Luo & Lai Ma & Pablo Lucas & Flaminio Squazzoni & Ana Marušić & Kalpana Shankar, 2019. "A scoping review of simulation models of peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 555-594, October.
    2. Michail Kovanis & Ludovic Trinquart & Philippe Ravaud & Raphaël Porcher, 2017. "Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 651-671, October.
    3. Bravo, Giangiacomo & Farjam, Mike & Grimaldo Moreno, Francisco & Birukou, Aliaksandr & Squazzoni, Flaminio, 2018. "Hidden connections: Network effects on editorial decisions in four computer science journals," Journal of Informetrics, Elsevier, vol. 12(1), pages 101-112.
    4. Federico Bianchi & Francisco Grimaldo & Giangiacomo Bravo & Flaminio Squazzoni, 2018. "The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1401-1420, September.
    5. Simone Righi & Károly Takács, 2017. "The miracle of peer review and development in science: an agent-based model," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 587-607, October.
    6. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2021. "The interplay between the reviewer’s incentives and the journal’s quality standard," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3041-3061, April.
    7. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "On the interplay between normalisation, bias, and performance of paper impact metrics," Journal of Informetrics, Elsevier, vol. 13(1), pages 270-290.
    8. Loet Leydesdorff, 2015. "Can intellectual processes in the sciences also be simulated? The anticipation and visualization of possible future states," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 2197-2214, December.
    9. Mario Paolucci & Francisco Grimaldo, 2014. "Mechanism change in a simulation of peer review: from junk support to elitism," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(3), pages 663-688, June.
    10. Matthias Meyer, 2011. "Bibliometrics, Stylized Facts and the Way Ahead: How to Build Good Social Simulation Models of Science?," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 14(4), pages 1-4.
    11. Isager, Peder Mortvedt & van 't Veer, Anna Elisabeth & Lakens, Daniel, 2021. "Replication value as a function of citation impact and sample size," MetaArXiv knjea, Center for Open Science.
    12. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2020. "The author–reviewer game," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2409-2431, September.
    13. Pawel Sobkowicz, 2015. "Innovation Suppression and Clique Evolution in Peer-Review-Based, Competitive Research Funding Systems: An Agent-Based Model," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 18(2), pages 1-13.
    14. Katchanov, Yurij L. & Markova, Yulia V. & Shmatko, Natalia A., 2023. "Uncited papers in the structure of scientific communication," Journal of Informetrics, Elsevier, vol. 17(2).
    15. David Chavalarias, 2017. "What’s wrong with Science?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 481-503, January.
    16. Zhang, Guangyao & Xu, Shenmeng & Sun, Yao & Jiang, Chunlin & Wang, Xianwen, 2022. "Understanding the peer review endeavor in scientific publishing," Journal of Informetrics, Elsevier, vol. 16(2).
    17. Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 695-715, February.
    18. Thomas Feliciani & Ramanathan Moorthy & Pablo Lucas & Kalpana Shankar, 2020. "Grade Language Heterogeneity in Simulation Models of Peer Review," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 23(3), pages 1-8.
    19. Carayol, Nicolas & Dalle, Jean-Michel, 2007. "Sequential problem choice and the reward system in Open Science," Structural Change and Economic Dynamics, Elsevier, vol. 18(2), pages 167-191, June.
    20. Ramona Weinrich, 2019. "Opportunities for the Adoption of Health-Based Sustainable Dietary Patterns: A Review on Consumer Research of Meat Substitutes," Sustainability, MDPI, vol. 11(15), pages 1-15, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:116:y:2018:i:3:d:10.1007_s11192-018-2826-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.