IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v104y2015i1d10.1007_s11192-015-1577-7.html
   My bibliography  Save this article

Qualifying the performance evaluation of Big Science beyond productivity, impact and costs

Author

Listed:
  • Richard Heidler

    (Wuppertal University)

  • Olof Hallonsten

    (Lund University)

Abstract

The use of quantitative performance measures to evaluate the productivity, impact and quality of research has spread to almost all parts of public R&D systems, including Big Science where traditional measures of technical reliability of instruments and user oversubscription have been joined by publication counts to assess scientific productivity. But such performance assessment has been shown to lead to absurdities, as the calculated average cost of single journal publications easily may reach hundreds of millions of dollars. In this article, the issue of productivity and impact is therefore further qualified by the use of additional measures such as the immediacy index as well as network analysis to evaluate qualitative aspects of the impact of contemporary Big Science labs. Connecting to previous work within what has been called “facilitymetrics”, the article continues the search for relevant bibliometric measures of the performance of Big Science labs with the use of a case study of a recently opened facility that is advertised as contributing to “breakthrough” research, by using several more measures and thus qualifying the topic of performance evaluation in contemporary Big Science beyond simple counts of publications, citations, and costs.

Suggested Citation

  • Richard Heidler & Olof Hallonsten, 2015. "Qualifying the performance evaluation of Big Science beyond productivity, impact and costs," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(1), pages 295-312, July.
    • Handle: RePEc:spr:scient:v:104:y:2015:i:1:d:10.1007_s11192-015-1577-7
    DOI: 10.1007/s11192-015-1577-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-015-1577-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-015-1577-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Aant Elzinga, 2012. "Features of the current science policy regime: Viewed in historical perspective," Science and Public Policy, Oxford University Press, vol. 39(4), pages 416-428, August.
    2. Thomas Heinze & Richard Heidler & Raphael Heiko Heiberger & Jan Riebling, 2013. "New patterns of scientific growth: How research expanded after the invention of scanning tunneling microscopy and the discovery of Buckminsterfullerenes," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(4), pages 829-843, April.
    3. Olof Hallonsten & Thomas Heinze, 2012. "Institutional persistence through gradual organizational adaptation: Analysis of national laboratories in the USA and Germany," Science and Public Policy, Oxford University Press, vol. 39(4), pages 450-463, July.
    4. Henry Small, 2010. "Maps of science as interdisciplinary discourse: co-citation contexts and the role of analogy," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(3), pages 835-849, June.
    5. Olof Hallonsten, 2014. "How expensive is Big Science? Consequences of using simple publication counts in performance assessment of large scientific facilities," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(2), pages 483-496, August.
    6. Olof Hallonsten, 2013. "Introducing ‘facilitymetrics’: a first review and analysis of commonly used measures of scientific leadership among synchrotron radiation facilities worldwide," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 497-513, August.
    7. Thomas Heinze & Richard Heidler & Raphael Heiko Heiberger & Jan Riebling, 2013. "New patterns of scientific growth: How research expanded after the invention of scanning tunneling microscopy and the discovery of Buckminsterfullerenes," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(4), pages 829-843, April.
    8. Alan L. Porter & Ismael Rafols, 2009. "Is science becoming more interdisciplinary? Measuring and mapping six research fields over time," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 719-745, December.
    9. Michael Gowanlock & Rich Gazan, 2013. "Assessing researcher interdisciplinarity: a case study of the University of Hawaii NASA Astrobiology Institute," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(1), pages 133-161, January.
    10. Catherine Westfall, 2012. "Institutional persistence and the material transformation of the US national labs: The curious story of the advent of the Advanced Photon Source," Science and Public Policy, Oxford University Press, vol. 39(4), pages 439-449, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Qiao, Lili & Mu, Rongping & Chen, Kaihua, 2016. "Scientific effects of large research infrastructures in China," Technological Forecasting and Social Change, Elsevier, vol. 112(C), pages 102-112.
    2. Wenchao Xu & Yanmei Xu & Junfeng Li, 2017. "A Study of RI Clusters Based on Symbiosis Theory," Sustainability, MDPI, vol. 9(3), pages 1-13, March.
    3. D’Ippolito, Beatrice & Rüling, Charles-Clemens, 2019. "Research collaboration in Large Scale Research Infrastructures: Collaboration types and policy implications," Research Policy, Elsevier, vol. 48(5), pages 1282-1296.
    4. Pan, Xiongfeng & Guo, Shucen & Li, Mengna & Song, Jinbo, 2021. "The effect of technology infrastructure investment on technological innovation ——A study based on spatial durbin model," Technovation, Elsevier, vol. 107(C).
    5. Xiyi Yang & Xiaoyu Zhou & Cong Cao, 2024. "Beamtimes and knowledge production times: how big-science research infrastructures shape nations’ domestic and international science production," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-11, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Olof Hallonsten, 2014. "How expensive is Big Science? Consequences of using simple publication counts in performance assessment of large scientific facilities," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(2), pages 483-496, August.
    2. Olof Hallonsten, 2013. "Introducing ‘facilitymetrics’: a first review and analysis of commonly used measures of scientific leadership among synchrotron radiation facilities worldwide," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 497-513, August.
    3. D’Ippolito, Beatrice & Rüling, Charles-Clemens, 2019. "Research collaboration in Large Scale Research Infrastructures: Collaboration types and policy implications," Research Policy, Elsevier, vol. 48(5), pages 1282-1296.
    4. Zuo, Zhiya & Zhao, Kang, 2018. "The more multidisciplinary the better? – The prevalence and interdisciplinarity of research collaborations in multidisciplinary institutions," Journal of Informetrics, Elsevier, vol. 12(3), pages 736-756.
    5. Thomas Heinze & Joel Emanuel Fuchs, 2022. "National and organizational patterns of Nobel laureate careers in physiology/medicine, physics, and chemistry," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7273-7288, December.
    6. Kristofer Rolf Söderström, 2023. "The structure and dynamics of instrument collaboration networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3581-3600, June.
    7. Zehra Taşkın & Arsev U. Aydinoglu, 2015. "Collaborative interdisciplinary astrobiology research: a bibliometric study of the NASA Astrobiology Institute," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 1003-1022, June.
    8. Xuefeng Wang & Zhinan Wang & Ying Huang & Yun Chen & Yi Zhang & Huichao Ren & Rongrong Li & Jinhui Pang, 2017. "Measuring interdisciplinarity of a research system: detecting distinction between publication categories and citation categories," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 2023-2039, June.
    9. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).
    10. Michael Gowanlock & Rich Gazan, 2013. "Assessing researcher interdisciplinarity: a case study of the University of Hawaii NASA Astrobiology Institute," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(1), pages 133-161, January.
    11. Finn Valentin & Maria Theresa Norn & Lars Alkaersig, 2016. "Orientations and outcome of interdisciplinary research: the case of research behaviour in translational medical science," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 67-90, January.
    12. Kamilla Kohn Rådberg & Hans Löfsten, 2024. "The entrepreneurial university and development of large-scale research infrastructure: exploring the emerging university function of collaboration and leadership," The Journal of Technology Transfer, Springer, vol. 49(1), pages 334-366, February.
    13. Young-Sun Jang & Young Joo Ko, 2019. "How latecomers catch up to leaders in high-energy physics as Big Science: transition from national system to international collaboration," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 437-480, April.
    14. Sichao Tong & Per Ahlgren, 2017. "Evolution of three Nobel Prize themes and a Nobel snub theme in chemistry: a bibliometric study with focus on international collaboration," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 75-90, July.
    15. Sander Zwanenburg & Maryam Nakhoda & Peter Whigham, 2022. "Toward greater consistency and validity in measuring interdisciplinarity: a systematic and conceptual evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7769-7788, December.
    16. Ryo Takahashi & Kenji Kaibe & Kazuyuki Suzuki & Sayaka Takahashi & Kotaro Takeda & Marc Hansen & Michiaki Yumoto, 2023. "New concept of the affinity between research fields using academic journal data in Scopus," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3507-3534, June.
    17. Wenchao Xu & Yanmei Xu & Junfeng Li, 2017. "A Study of RI Clusters Based on Symbiosis Theory," Sustainability, MDPI, vol. 9(3), pages 1-13, March.
    18. Kyle Siler & Philippe Vincent-Lamarre & Cassidy R Sugimoto & Vincent Larivière, 2022. "Cumulative advantage and citation performance of repeat authors in scholarly journals," PLOS ONE, Public Library of Science, vol. 17(4), pages 1-17, April.
    19. Stephen Carley & Alan L. Porter, 2012. "A forward diversity index," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(2), pages 407-427, February.
    20. Jian Xu & Yi Bu & Ying Ding & Sinan Yang & Hongli Zhang & Chen Yu & Lin Sun, 2018. "Understanding the formation of interdisciplinary research from the perspective of keyword evolution: a case study on joint attention," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 973-995, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:104:y:2015:i:1:d:10.1007_s11192-015-1577-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.