IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v127y2022i12d10.1007_s11192-022-04413-7.html
   My bibliography  Save this article

Identifying benchmark units for research management and evaluation

Author

Listed:
  • Qi Wang

    (KTH-Royal Institute of Technology, KTH Library & Division of History of Science)

  • Tobias Jeppsson

    (KTH-Royal Institute of Technology, KTH Library)

Abstract

While normalized bibliometric indicators are expected to resolve the subject-field differences between organizations in research evaluations, the identification of reference organizations working on similar research topics is still of importance. Research organizations, policymakers and research funders tend to use benchmark units as points of comparison for a certain research unit in order to understand and monitor its development and performance. In addition, benchmark organizations can also be used to pinpoint potential collaboration partners or competitors. Therefore, methods for identifying benchmark research units are of practical significance. Even so, few studies have further explored this problem. This study aims to propose a bibliometric approach for the identification of benchmark units. We define an appropriate benchmark as a well-connected research environment, in which researchers investigate similar topics and publish a similar number of publications compared to a given research organization during the same period. Four essential attributes for the evaluation of benchmarks are research topics, output, connectedness, and scientific impact. We apply this strategy to two research organizations in Sweden and examine the effectiveness of the proposed method. Identified benchmark units are evaluated by examining the research similarity and the robustness of various measures of connectivity.

Suggested Citation

  • Qi Wang & Tobias Jeppsson, 2022. "Identifying benchmark units for research management and evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7557-7574, December.
    • Handle: RePEc:spr:scient:v:127:y:2022:i:12:d:10.1007_s11192-022-04413-7
    DOI: 10.1007/s11192-022-04413-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-022-04413-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-022-04413-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Henning, Theunis F.P. & Muruvan, Sugandree & Feng, Wanhua A. & Dunn, Roger C., 2011. "The development of a benchmarking tool for monitoring progress towards sustainable transportation in New Zealand," Transport Policy, Elsevier, vol. 18(2), pages 480-488, March.
    2. Hougaard, Jens Leth & Tvede, Mich, 2002. "Benchmark selection: An axiomatic approach," European Journal of Operational Research, Elsevier, vol. 137(1), pages 218-228, February.
    3. Frenken, Koen & Heimeriks, Gaston J. & Hoekman, Jarno, 2017. "What drives university research performance? An analysis using the CWTS Leiden Ranking data," Journal of Informetrics, Elsevier, vol. 11(3), pages 859-872.
    4. E.C.M. Noyons & H.F. Moed & M. Luwel, 1999. "Combining mapping and citation analysis for evaluative bibliometric purposes: A bibliometric study," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 50(2), pages 115-131.
    5. Ruiz-Castillo, Javier & Waltman, Ludo, 2015. "Field-normalized citation impact indicators using algorithmically constructed classification systems of science," Journal of Informetrics, Elsevier, vol. 9(1), pages 102-117.
    6. Ludo Waltman & Nees Jan van Eck, 2012. "A new methodology for constructing a publication‐level classification system of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    7. Ludo Waltman & Nees Eck, 2013. "A smart local moving algorithm for large-scale modularity-based community detection," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 86(11), pages 1-14, November.
    8. Ludo Waltman & Michael Schreiber, 2013. "On the calculation of percentile-based bibliometric indicators," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 372-379, February.
    9. Nees Jan Eck & Ludo Waltman, 2010. "Software survey: VOSviewer, a computer program for bibliometric mapping," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(2), pages 523-538, August.
    10. Lucertini, M. & Nicolo, F. & Telmon, D., 1995. "Integration of benchmarking and benchmarking of integration," International Journal of Production Economics, Elsevier, vol. 38(1), pages 59-71, March.
    11. Ludo Waltman & Nees Jan Eck, 2012. "A new methodology for constructing a publication-level classification system of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    12. Grün, Bettina & Hornik, Kurt, 2011. "topicmodels: An R Package for Fitting Topic Models," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 40(i13).
    13. Li, Yongjun & Liang, Liang & Chen, Yao & Morita, Hiroshi, 2008. "Models for measuring and benchmarking olympics achievements," Omega, Elsevier, vol. 36(6), pages 933-940, December.
    14. Michel Zitt & Suzy Ramanana-Rahary & Elise Bassecoulard, 2005. "Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(2), pages 373-401, April.
    15. Nicolas Carayol & Ghislaine Filliatreau & Agenor Lahatte, 2012. "Reference classes: a tool for benchmarking universities’ research," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(2), pages 351-371, November.
    16. Ludo Waltman & Michael Schreiber, 2013. "On the calculation of percentile‐based bibliometric indicators," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 372-379, February.
    17. Qi Wang, 2018. "A bibliometric model for identifying emerging research topics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 69(2), pages 290-304, February.
    18. J Sylvan Katz, 2000. "Scale-independent indicators and research evaluation," Science and Public Policy, Oxford University Press, vol. 27(1), pages 23-36, February.
    19. Tuzkaya, Gülfem & Sennaroglu, Bahar & Kalender, Zeynep Tuğçe & Mutlu, Meltem, 2019. "Hospital service quality evaluation with IVIF-PROMETHEE and a case study," Socio-Economic Planning Sciences, Elsevier, vol. 68(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lutz Bornmann & Benedetto Lepori, 2024. "The use of ChatGPT to find similar institutions for institutional benchmarking," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(6), pages 3593-3598, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "Globalised vs averaged: Bias and ranking performance on the author level," Journal of Informetrics, Elsevier, vol. 13(1), pages 299-313.
    2. Ruiz-Castillo, Javier & Waltman, Ludo, 2015. "Field-normalized citation impact indicators using algorithmically constructed classification systems of science," Journal of Informetrics, Elsevier, vol. 9(1), pages 102-117.
    3. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    4. Mariana-Daniela González-Zamar & Emilio Abad-Segura, 2021. "Emotional Creativity in Art Education: An Exploratory Analysis and Research Trends," IJERPH, MDPI, vol. 18(12), pages 1-20, June.
    5. Bornmann, Lutz & Haunschild, Robin, 2022. "Empirical analysis of recent temporal dynamics of research fields: Annual publications in chemistry and related areas as an example," Journal of Informetrics, Elsevier, vol. 16(2).
    6. Francesco Pasimeni, 2020. "The Origin of the Sharing Economy Meets the Legacy of Fractional Ownership," SPRU Working Paper Series 2020-19, SPRU - Science Policy Research Unit, University of Sussex Business School.
    7. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    8. Leydesdorff, Loet & Bornmann, Lutz & Zhou, Ping, 2016. "Construction of a pragmatic base line for journal classifications and maps based on aggregated journal-journal citation relations," Journal of Informetrics, Elsevier, vol. 10(4), pages 902-918.
    9. van Eck, Nees Jan & Waltman, Ludo, 2014. "CitNetExplorer: A new software tool for analyzing and visualizing citation networks," Journal of Informetrics, Elsevier, vol. 8(4), pages 802-823.
    10. Nees Jan Eck & Ludo Waltman, 2017. "Citation-based clustering of publications using CitNetExplorer and VOSviewer," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(2), pages 1053-1070, May.
    11. Zang, Yuzhu & Yang, Yuanyuan & Liu, Yansui, 2021. "Toward serving land consolidation on the table of sustainability: An overview of the research landscape and future directions," Land Use Policy, Elsevier, vol. 109(C).
    12. Paola Bernardi & Alberto Bertello & Canio Forliano & Ludovico Bullini Orlandi, 2022. "Beyond the “ivory tower”. Comparing academic and non-academic knowledge on social entrepreneurship," International Entrepreneurship and Management Journal, Springer, vol. 18(3), pages 999-1032, September.
    13. Anthony F. J. Raan, 2021. "Laudation on the occasion of the presentation of the Derek de Solla Price Award 2021 to Prof. Ludo Waltman at the ISSI conference, Leuven, 2021," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(10), pages 8235-8238, October.
    14. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    15. Bouyssou, Denis & Marchant, Thierry, 2016. "Ranking authors using fractional counting of citations: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 183-199.
    16. Xu, Haiyun & Winnink, Jos & Yue, Zenghui & Zhang, Huiling & Pang, Hongshen, 2021. "Multidimensional Scientometric indicators for the detection of emerging research topics," Technological Forecasting and Social Change, Elsevier, vol. 163(C).
    17. Forliano, Canio & De Bernardi, Paola & Yahiaoui, Dorra, 2021. "Entrepreneurial universities: A bibliometric analysis within the business and management domains," Technological Forecasting and Social Change, Elsevier, vol. 165(C).
    18. Antonio Perianes-Rodriguez & Javier Ruiz-Castillo, 2016. "A comparison of two ways of evaluating research units working in different scientific fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 539-561, February.
    19. Shuo Xu & Junwan Liu & Dongsheng Zhai & Xin An & Zheng Wang & Hongshen Pang, 2018. "Overlapping thematic structures extraction with mixed-membership stochastic blockmodel," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 61-84, October.
    20. Waltman, Ludo & van Eck, Nees Jan, 2013. "A systematic empirical comparison of different approaches for normalizing citation impact indicators," Journal of Informetrics, Elsevier, vol. 7(4), pages 833-849.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:127:y:2022:i:12:d:10.1007_s11192-022-04413-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.