IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v126y2021i8d10.1007_s11192-021-04049-z.html
   My bibliography  Save this article

Introducing a novelty indicator for scientific research: validating the knowledge-based combinatorial approach

Author

Listed:
  • Kuniko Matsumoto

    (National Institute of Science and Technology Policy (NISTEP))

  • Sotaro Shibayama

    (Lund University)

  • Byeongwoo Kang

    (Hitotsubashi University)

  • Masatsura Igami

    (National Institute of Science and Technology Policy (NISTEP))

Abstract

Citation counts have long been considered as the primary bibliographic indicator for evaluating the quality of research—a practice premised on the assumption that citation count is reflective of the impact of a scientific publication. However, identifying several limitations in the use of citation counts alone, scholars have advanced the need for multifaceted quality evaluation methods. In this study, we apply a novelty indicator to quantify the degree of citation similarity between a focal paper and a pre-existing same-domain paper from various fields in the natural sciences by proposing a new way of identifying papers that fall into the same domain of focal papers using bibliometric data only. We also conduct a validation analysis, using Japanese survey data, to confirm its usefulness. Employing ordered logit and ordinary least squares regression models, this study tests the consistency between the novelty scores of 1871 Japanese papers published in the natural sciences between 2001 and 2006 and researchers’ subjective judgments of their novelty. The results show statistically positive correlations between novelty scores and researchers’ assessment of research types reflecting aspects of novelty in various natural science fields. As such, this study demonstrates that the proposed novelty indicator is a suitable means of identifying the novelty of various types of natural scientific research.

Suggested Citation

  • Kuniko Matsumoto & Sotaro Shibayama & Byeongwoo Kang & Masatsura Igami, 2021. "Introducing a novelty indicator for scientific research: validating the knowledge-based combinatorial approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 6891-6915, August.
    • Handle: RePEc:spr:scient:v:126:y:2021:i:8:d:10.1007_s11192-021-04049-z
    DOI: 10.1007/s11192-021-04049-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-021-04049-z
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-021-04049-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Walsh, John P. & Lee, You-Na, 2015. "The bureaucratization of science," Research Policy, Elsevier, vol. 44(8), pages 1584-1600.
    2. Dahlin, Kristina B. & Behrens, Dean M., 2005. "When is an invention really radical?: Defining and measuring technological radicalness," Research Policy, Elsevier, vol. 34(5), pages 717-737, June.
    3. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    4. Bornmann, Lutz & Tekles, Alexander & Zhang, Helena H. & Ye, Fred Y., 2019. "Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data," Journal of Informetrics, Elsevier, vol. 13(4).
    5. Verhoeven, Dennis & Bakker, Jurriën & Veugelers, Reinhilde, 2016. "Measuring technological novelty with patent-based indicators," Research Policy, Elsevier, vol. 45(3), pages 707-723.
    6. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    7. Trapido, Denis, 2015. "How novelty in knowledge earns recognition: The role of consistent identities," Research Policy, Elsevier, vol. 44(8), pages 1488-1500.
    8. Tahamtan, Iman & Bornmann, Lutz, 2018. "Creativity in science and the link to cited references: Is the creative potential of papers reflected in their cited references?," Journal of Informetrics, Elsevier, vol. 12(3), pages 906-930.
    9. Kristina Dahlin & Deans M. Behrens, 2005. "When is an invention really radical? Defining and measuring technological radicalness," Post-Print hal-00480416, HAL.
    10. Sarah Kaplan & Keyvan Vakili, 2015. "The double-edged sword of recombination in breakthrough innovation," Strategic Management Journal, Wiley Blackwell, vol. 36(10), pages 1435-1457, October.
    11. Murayama, Kota & Nirei, Makoto & Shimizu, Hiroshi, 2015. "Management of science, serendipity, and research performance: Evidence from a survey of scientists in Japan and the U.S," Research Policy, Elsevier, vol. 44(4), pages 862-873.
    12. Lee, You-Na & Walsh, John P. & Wang, Jian, 2015. "Creativity in scientific teams: Unpacking novelty and impact," Research Policy, Elsevier, vol. 44(3), pages 684-697.
    13. Nagaoka, Sadao & 長岡, 貞男 & Igami, Masatsura & 伊神, 正貫 & Eto, Manabu & 江藤, 学 & Ijichi, Tomohiro & 伊地知, 寛博, 2010. "Knowledge Creation Process in Science : Basic findings from a large‐scale survey of researchers in Japan," IIR Working Paper 10-08, Institute of Innovation Research, Hitotsubashi University.
    14. Bornmann, Lutz & Schier, Hermann & Marx, Werner & Daniel, Hans-Dieter, 2012. "What factors determine citation counts of publications in chemistry besides their quality?," Journal of Informetrics, Elsevier, vol. 6(1), pages 11-18.
    15. Masatsura Igami & Sadao Nagaoka & John Walsh, 2015. "Contribution of postdoctoral fellows to fast-moving and competitive scientific research," The Journal of Technology Transfer, Springer, vol. 40(4), pages 723-741, August.
    16. Charles Oppenheim & Susan P. Renn, 1978. "Highly cited old papers and the reasons why they continue to be cited," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 29(5), pages 225-231, September.
    17. Tanzila Ahmed & Ben Johnson & Charles Oppenheim & Catherine Peck, 2004. "Highly cited old papers and the reasons why they continue to be cited. Part II., The 1953 Watson and Crick article on the structure of DNA," Scientometrics, Springer;Akadémiai Kiadó, vol. 61(2), pages 147-156, October.
    18. Wang, Jian & Lee, You-Na & Walsh, John P., 2018. "Funding model and creativity in science: Competitive versus block funding and status contingency effects," Research Policy, Elsevier, vol. 47(6), pages 1070-1083.
    19. Lee Fleming, 2001. "Recombinant Uncertainty in Technological Search," Management Science, INFORMS, vol. 47(1), pages 117-132, January.
    20. Paul M. Romer, 1994. "The Origins of Endogenous Growth," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 3-22, Winter.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jeon, Daeseong & Lee, Junyoup & Ahn, Joon Mo & Lee, Changyong, 2023. "Measuring the novelty of scientific publications: A fastText and local outlier factor approach," Journal of Informetrics, Elsevier, vol. 17(4).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dongqing Lyu & Kaile Gong & Xuanmin Ruan & Ying Cheng & Jiang Li, 2021. "Does research collaboration influence the “disruption” of articles? Evidence from neurosciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 287-303, January.
    2. Bornmann, Lutz & Tekles, Alexander & Zhang, Helena H. & Ye, Fred Y., 2019. "Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data," Journal of Informetrics, Elsevier, vol. 13(4).
    3. Sotaro Shibayama & Deyun Yin & Kuniko Matsumoto, 2021. "Measuring novelty in science with word embedding," PLOS ONE, Public Library of Science, vol. 16(7), pages 1-16, July.
    4. Ron Boschma & Ernest Miguelez & Rosina Moreno & Diego B. Ocampo-Corrales, 2021. "Technological breakthroughs in European regions: the role of related and unrelated combinations," Papers in Evolutionary Economic Geography (PEEG) 2118, Utrecht University, Department of Human Geography and Spatial Planning, Group Economic Geography, revised Jun 2021.
    5. Yan Yan & Shanwu Tian & Jingjing Zhang, 2020. "The impact of a paper’s new combinations and new components on its citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 895-913, February.
    6. Kathryn Rudie Harrigan & Maria Chiara Guardo & Bo Cowgill, 2017. "Multiplicative-innovation synergies: tests in technological acquisitions," The Journal of Technology Transfer, Springer, vol. 42(5), pages 1212-1233, October.
    7. Ugo Rizzo & Nicolò Barbieri & Laura Ramaciotti & Demian Iannantuono, 2020. "The division of labour between academia and industry for the generation of radical inventions," The Journal of Technology Transfer, Springer, vol. 45(2), pages 393-413, April.
    8. Deichmann, Dirk & Moser, Christine & Birkholz, Julie M. & Nerghes, Adina & Groenewegen, Peter & Wang, Shenghui, 2020. "Ideas with impact: How connectivity shapes idea diffusion," Research Policy, Elsevier, vol. 49(1).
    9. Qu, Guannan & Chen, Jin & Zhang, Ruhao & Wang, Luyao & Yang, Yayu, 2023. "Technological search strategy and breakthrough innovation: An integrated approach based on main-path analysis," Technological Forecasting and Social Change, Elsevier, vol. 196(C).
    10. Sandro Montresor & Gianluca Orsatti & Francesco Quatraro, 2023. "Technological novelty and key enabling technologies: evidence from European regions," Economics of Innovation and New Technology, Taylor & Francis Journals, vol. 32(6), pages 851-872, August.
    11. Doblinger, Claudia & Surana, Kavita & Li, Deyu & Hultman, Nathan & Anadón, Laura Díaz, 2022. "How do global manufacturing shifts affect long-term clean energy innovation? A study of wind energy suppliers," Research Policy, Elsevier, vol. 51(7).
    12. Michele Cincera & Ela Ince, 2019. "Types of Innovation and Firm performance," Working Papers TIMES² 2019-032, ULB -- Universite Libre de Bruxelles.
    13. Hou, Jianhua & Wang, Dongyi & Li, Jing, 2022. "A new method for measuring the originality of academic articles based on knowledge units in semantic networks," Journal of Informetrics, Elsevier, vol. 16(3).
    14. Brea, Edgar, 2024. "The yin yang of AI: Exploring how commercial and non-commercial orientations shape machine learning innovation," Research Policy, Elsevier, vol. 53(6).
    15. Sun, Bixuan & Kolesnikov, Sergey & Goldstein, Anna & Chan, Gabriel, 2021. "A dynamic approach for identifying technological breakthroughs with an application in solar photovoltaics," Technological Forecasting and Social Change, Elsevier, vol. 165(C).
    16. Fontana, Magda & Iori, Martina & Montobbio, Fabio & Sinatra, Roberta, 2020. "New and atypical combinations: An assessment of novelty and interdisciplinarity," Research Policy, Elsevier, vol. 49(7).
    17. Luo, Zhuoran & Lu, Wei & He, Jiangen & Wang, Yuqi, 2022. "Combination of research questions and methods: A new measurement of scientific novelty," Journal of Informetrics, Elsevier, vol. 16(2).
    18. Dirk Fornahl & Nils Grashof & Alexander Kopka, 2021. "Do not neglect the periphery?! - the emergence and diffusion of radical innovations," Bremen Papers on Economics & Innovation 2102, University of Bremen, Faculty of Business Studies and Economics.
    19. Mariia Shkolnykova & Muhamed Kudic, 2022. "Who benefits from SMEs’ radical innovations?—empirical evidence from German biotechnology," Small Business Economics, Springer, vol. 58(2), pages 1157-1185, February.
    20. Veugelers, Reinhilde & Wang, Jian, 2019. "Scientific novelty and technological impact," Research Policy, Elsevier, vol. 48(6), pages 1362-1372.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:126:y:2021:i:8:d:10.1007_s11192-021-04049-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.