IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v109y2016i3d10.1007_s11192-016-2132-x.html
   My bibliography  Save this article

Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level

Author

Listed:
  • A. I. M. Jakaria Rahman

    (University of Antwerp)

  • Raf Guns

    (University of Antwerp)

  • Loet Leydesdorff

    (University of Amsterdam)

  • Tim C. E. Engels

    (University of Antwerp
    Antwerp Maritime Academy)

Abstract

When research groups are evaluated by an expert panel, it is an open question how one can determine the match between panel and research groups. In this paper, we outline two quantitative approaches that determine the cognitive distance between evaluators and evaluees, based on the journals they have published in. We use example data from four research evaluations carried out between 2009 and 2014 at the University of Antwerp. While the barycenter approach is based on a journal map, the similarity-adapted publication vector (SAPV) approach is based on the full journal similarity matrix. Both approaches determine an entity’s profile based on the journals in which it has published. Subsequently, we determine the Euclidean distance between the barycenter or SAPV profiles of two entities as an indicator of the cognitive distance between them. Using a bootstrapping approach, we determine confidence intervals for these distances. As such, the present article constitutes a refinement of a previous proposal that operates on the level of Web of Science subject categories.

Suggested Citation

  • A. I. M. Jakaria Rahman & Raf Guns & Loet Leydesdorff & Tim C. E. Engels, 2016. "Measuring the match between evaluators and evaluees: cognitive distances between panel members and research groups at the journal level," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1639-1663, December.
    • Handle: RePEc:spr:scient:v:109:y:2016:i:3:d:10.1007_s11192-016-2132-x
    DOI: 10.1007/s11192-016-2132-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-016-2132-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-016-2132-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Pawel Sobkowicz, 2015. "Innovation Suppression and Clique Evolution in Peer-Review-Based, Competitive Research Funding Systems: An Agent-Based Model," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 18(2), pages 1-13.
    2. Yuen-Hsien Tseng & Ming-Yueh Tsay, 2013. "Journal clustering of library and information science for subfield delineation using the bibliometric analysis toolkit: CATAR," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(2), pages 503-528, May.
    3. Ludo Waltman & Nees Jan van Eck, 2012. "A new methodology for constructing a publication‐level classification system of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    4. Nooteboom, Bart & Van Haverbeke, Wim & Duysters, Geert & Gilsing, Victor & van den Oord, Ad, 2007. "Optimal cognitive distance and absorptive capacity," Research Policy, Elsevier, vol. 36(7), pages 1016-1034, September.
    5. T. Gorjiara & C. Baldock, 2014. "Nanoscience and nanotechnology research publications: a comparison between Australia and the rest of the world," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(1), pages 121-148, July.
    6. Loet Leydesdorff & Gaston Heimeriks & Daniele Rotolo, 2016. "Journal portfolio analysis for countries, cities, and organizations: Maps and comparisons," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(3), pages 741-748, March.
    7. Verleysen, Frederik T. & Engels, Tim C.E., 2014. "Barycenter representation of book publishing internationalization in the Social Sciences and Humanities," Journal of Informetrics, Elsevier, vol. 8(1), pages 234-240.
    8. Finn Hansson, 2010. "Dialogue in or with the peer review? Evaluating research organizations in order to promote organizational learning," Science and Public Policy, Oxford University Press, vol. 37(4), pages 239-251, May.
    9. Nees Jan van Eck & Ludo Waltman & Rommert Dekker & Jan van den Berg, 2010. "A comparison of two techniques for bibliometric mapping: Multidimensional scaling and VOS," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(12), pages 2405-2416, December.
    10. Shiji Chen & Clément Arsenault & Yves Gingras & Vincent Larivière, 2015. "Exploring the interdisciplinary evolution of a discipline: the case of Biochemistry and Molecular Biology," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1307-1323, February.
    11. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    12. Katharine Barker, 2007. "The UK Research Assessment Exercise: the evolution of a national research evaluation system," Research Evaluation, Oxford University Press, vol. 16(1), pages 3-12, March.
    13. Ludo Waltman & Nees Jan Eck, 2012. "A new methodology for constructing a publication-level classification system of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    14. Chris Fields, 2015. "How small is the center of science? Short cross-disciplinary cycles in co-authorship graphs," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1287-1306, February.
    15. Leydesdorff, Loet & Rafols, Ismael, 2012. "Interactive overlays: A new method for generating global journal maps from Web-of-Science data," Journal of Informetrics, Elsevier, vol. 6(2), pages 318-332.
    16. Richard Berendsen & Maarten Rijke & Krisztian Balog & Toine Bogers & Antal Bosch, 2013. "On the assessment of expertise profiles," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(10), pages 2024-2044, October.
    17. Lutz Bornmann & Rüdiger Mutz & Werner Marx & Hermann Schier & Hans‐Dieter Daniel, 2011. "A multilevel modelling approach to investigating the predictive validity of editorial decisions: do the editors of a high profile journal select manuscripts that are highly cited after publication?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(4), pages 857-879, October.
    18. N Rons & A De Bruyn & J Cornelis, 2008. "Research evaluation per discipline: a peer-review method and its outcomes," Research Evaluation, Oxford University Press, vol. 17(1), pages 45-57, March.
    19. Richard Berendsen & Maarten de Rijke & Krisztian Balog & Toine Bogers & Antal van den Bosch, 2013. "On the assessment of expertise profiles," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(10), pages 2024-2044, October.
    20. Loet Leydesdorff & Ismael Rafols & Chaomei Chen, 2013. "Interactive overlays of journals and the measurement of interdisciplinarity on the basis of aggregated journal–journal citations," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(12), pages 2573-2586, December.
    21. Ismael Rafols & Alan L. Porter & Loet Leydesdorff, 2010. "Science overlay maps: A new tool for research policy and library management," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(9), pages 1871-1887, September.
    22. David G Pina & Darko Hren & Ana Marušić, 2015. "Peer Review Evaluation Process of Marie Curie Actions under EU’s Seventh Framework Programme for Research," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-15, June.
    23. Sebastian Grauwin & Pablo Jensen, 2011. "Mapping scientific institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 943-954, December.
    24. Peter van den Besselaar & Loet Leydesdorff, 2009. "Past performance, peer review and project selection: a case study in the social and behavioral sciences," Research Evaluation, Oxford University Press, vol. 18(4), pages 273-288, October.
    25. Cohen, Wesley M & Levinthal, Daniel A, 1989. "Innovation and Learning: The Two Faces of R&D," Economic Journal, Royal Economic Society, vol. 99(397), pages 569-596, September.
    26. Rahman, A.I.M. Jakaria & Guns, Raf & Rousseau, Ronald & Engels, Tim C.E., 2015. "Is the expertise of evaluation panels congruent with the research interests of the research groups: A quantitative approach based on barycenters," Journal of Informetrics, Elsevier, vol. 9(4), pages 704-721.
    27. Kevin W. Boyack & Richard Klavans, 2014. "Creation of a highly detailed, dynamic, global model and map of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 670-685, April.
    28. Nees Jan Eck & Ludo Waltman, 2010. "Software survey: VOSviewer, a computer program for bibliometric mapping," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(2), pages 523-538, August.
    29. Katja Hofmann & Krisztian Balog & Toine Bogers & Maarten de Rijke, 2010. "Contextual factors for finding similar experts," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(5), pages 994-1014, May.
    30. Bart Nooteboom, 2000. "Learning by Interaction: Absorptive Capacity, Cognitive Distance and Governance," Journal of Management & Governance, Springer;Accademia Italiana di Economia Aziendale (AIDEA), vol. 4(1), pages 69-92, March.
    31. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    32. Katja Hofmann & Krisztian Balog & Toine Bogers & Maarten de Rijke, 2010. "Contextual factors for finding similar experts," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(5), pages 994-1014, May.
    33. Qi Wang & Ulf Sandström, 2015. "Defining the role of cognitive distance in the peer review process with an explorative study of a grant scheme in infection biology," Research Evaluation, Oxford University Press, vol. 24(3), pages 271-281.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rousseau, Ronald & Guns, Raf & Rahman, A.I.M. Jakaria & Engels, Tim C.E., 2017. "Measuring cognitive distance between publication portfolios," Journal of Informetrics, Elsevier, vol. 11(2), pages 583-594.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rahman, A.I.M. Jakaria & Guns, Raf & Rousseau, Ronald & Engels, Tim C.E., 2015. "Is the expertise of evaluation panels congruent with the research interests of the research groups: A quantitative approach based on barycenters," Journal of Informetrics, Elsevier, vol. 9(4), pages 704-721.
    2. Leydesdorff, Loet & Bornmann, Lutz & Zhou, Ping, 2016. "Construction of a pragmatic base line for journal classifications and maps based on aggregated journal-journal citation relations," Journal of Informetrics, Elsevier, vol. 10(4), pages 902-918.
    3. Kevin W. Boyack, 2017. "Thesaurus-based methods for mapping contents of publication sets," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(2), pages 1141-1155, May.
    4. Gómez-Núñez, Antonio J. & Batagelj, Vladimir & Vargas-Quesada, Benjamín & Moya-Anegón, Félix & Chinchilla-Rodríguez, Zaida, 2014. "Optimizing SCImago Journal & Country Rank classification by community detection," Journal of Informetrics, Elsevier, vol. 8(2), pages 369-383.
    5. Yanto Chandra, 2018. "Mapping the evolution of entrepreneurship as a field of research (1990–2013): A scientometric analysis," PLOS ONE, Public Library of Science, vol. 13(1), pages 1-24, January.
    6. María Pinto & Rosaura Fernández-Pascual & David Caballero-Mariscal & Dora Sales, 2020. "Information literacy trends in higher education (2006–2019): visualizing the emerging field of mobile information literacy," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1479-1510, August.
    7. Jielan Ding & Per Ahlgren & Liying Yang & Ting Yue, 2018. "Disciplinary structures in Nature, Science and PNAS: journal and country levels," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1817-1852, September.
    8. Xiancheng Li & Luca Verginer & Massimo Riccaboni & P. Panzarasa, 2022. "A network approach to expertise retrieval based on path similarity and credit allocation," Journal of Economic Interaction and Coordination, Springer;Society for Economic Science with Heterogeneous Interacting Agents, vol. 17(2), pages 501-533, April.
    9. Tomasz Ingram & Monika Wieczorek-Kosmala & Karel Hlaváček, 2023. "Organizational Resilience as a Response to the Energy Crisis: Systematic Literature Review," Energies, MDPI, vol. 16(2), pages 1-35, January.
    10. Boyack, Kevin W. & Klavans, Richard, 2014. "Including cited non-source items in a large-scale map of science: What difference does it make?," Journal of Informetrics, Elsevier, vol. 8(3), pages 569-580.
    11. Hird, Mackenzie D. & Pfotenhauer, Sebastian M., 2017. "How complex international partnerships shape domestic research clusters: Difference-in-difference network formation and research re-orientation in the MIT Portugal Program," Research Policy, Elsevier, vol. 46(3), pages 557-572.
    12. van Eck, Nees Jan & Waltman, Ludo, 2014. "CitNetExplorer: A new software tool for analyzing and visualizing citation networks," Journal of Informetrics, Elsevier, vol. 8(4), pages 802-823.
    13. Chiara Carusi & Giuseppe Bianchi, 2020. "A look at interdisciplinarity using bipartite scholar/journal networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 867-894, February.
    14. Sjögårde, Peter & Ahlgren, Per, 2018. "Granularity of algorithmically constructed publication-level classifications of research publications: Identification of topics," Journal of Informetrics, Elsevier, vol. 12(1), pages 133-152.
    15. S. P. J. M. Horbach & W. Halffman, 2019. "The ability of different peer review procedures to flag problematic publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 339-373, January.
    16. Loet Leydesdorff & Duncan Kushnir & Ismael Rafols, 2014. "Interactive overlay maps for US patent (USPTO) data based on International Patent Classification (IPC)," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 1583-1599, March.
    17. Daniele Rotolo & Ismael Rafols & Michael Hopkins & Loet Leydesdorff, 2014. "Scientometric Mapping as a Strategic Intelligence Tool for the Governance of Emerging Technologies," SPRU Working Paper Series 2014-10, SPRU - Science Policy Research Unit, University of Sussex Business School.
    18. Rons, Nadine, 2018. "Bibliometric approximation of a scientific specialty by combining key sources, title words, authors and references," Journal of Informetrics, Elsevier, vol. 12(1), pages 113-132.
    19. Shu, Fei & Julien, Charles-Antoine & Zhang, Lin & Qiu, Junping & Zhang, Jing & Larivière, Vincent, 2019. "Comparing journal and paper level classifications of science," Journal of Informetrics, Elsevier, vol. 13(1), pages 202-225.
    20. Carlos Olmeda-Gómez & Carlos Romá-Mateo & Maria-Antonia Ovalle-Perandones, 2019. "Overview of trends in global epigenetic research (2009–2017)," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1545-1574, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:109:y:2016:i:3:d:10.1007_s11192-016-2132-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.