IDEAS home Printed from https://ideas.repec.org/a/eee/tefoso/v146y2019icp673-686.html
   My bibliography  Save this article

Searching for new breakthroughs in science: How effective are computerised detection algorithms?

Author

Listed:
  • Winnink, J.J.
  • Tijssen, Robert J.W.
  • van Raan, A.F.J.

Abstract

In this study, we design, develop, implement and test an analytical framework and measurement model to detect scientific discoveries with ‘breakthrough’ characteristics. To do so, we have developed a series of computerised search algorithms that data mine large quantities of research publications. These algorithms facilitate early-stage detection of ‘breakout’ papers that emerge as highly cited and distinctive and are considered to be potential breakthroughs. Combining computer-aided data mining with decision heuristics, enabled us to assess structural changes within citation patterns with the international scientific literature. In our case studies, we applied a citation impact time window of 24–36 months after publication of each research paper.

Suggested Citation

  • Winnink, J.J. & Tijssen, Robert J.W. & van Raan, A.F.J., 2019. "Searching for new breakthroughs in science: How effective are computerised detection algorithms?," Technological Forecasting and Social Change, Elsevier, vol. 146(C), pages 673-686.
    • Handle: RePEc:eee:tefoso:v:146:y:2019:i:c:p:673-686
    DOI: 10.1016/j.techfore.2018.05.018
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0040162517318863
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.techfore.2018.05.018?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Richard Van Noorden, 2011. "Science publishing: The trouble with retractions," Nature, Nature, vol. 478(7367), pages 26-28, October.
    2. Luís M. A. Bettencourt & David I. Kaiser & Jasleen Kaur & Carlos Castillo-Chávez & David E. Wojick, 2008. "Population modeling of the emergence and development of scientific fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 75(3), pages 495-518, June.
    3. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    4. David Cyranoski, 2004. "Korea's stem-cell stars dogged by suspicion of ethical breach," Nature, Nature, vol. 429(6987), pages 3-3, May.
    5. Ludo Waltman & Nees Jan van Eck, 2012. "A new methodology for constructing a publication‐level classification system of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    6. Jesper W. Schneider & Rodrigo Costas, 2017. "Identifying potential “breakthrough” publications using refined citation analyses: Three related explorative approaches," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(3), pages 709-723, March.
    7. John Whitfield, 2008. "Collaboration: Group theory," Nature, Nature, vol. 455(7214), pages 720-723, October.
    8. Marc Julius & Charles E. Berkoff & Alvin E. Strack & Frank Krasovec & A. Douglas Bender, 1977. "A very early warning system for the rapid identification and transfer of new technology," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 28(3), pages 170-174, May.
    9. Marten Scheffer, 2010. "Foreseeing tipping points," Nature, Nature, vol. 467(7314), pages 411-412, September.
    10. Paul F. Skilton, 2009. "Does the human capital of teams of natural science authors predict citation frequency?," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(3), pages 525-542, March.
    11. Bettencourt, Luís M.A. & Kaiser, David I. & Kaur, Jasleen, 2009. "Scientific discovery and topological transitions in collaboration networks," Journal of Informetrics, Elsevier, vol. 3(3), pages 210-221.
    12. Ludo Waltman & Nees Jan Eck, 2012. "A new methodology for constructing a publication-level classification system of science," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(12), pages 2378-2392, December.
    13. Chen, Chaomei & Chen, Yue & Horowitz, Mark & Hou, Haiyan & Liu, Zeyuan & Pellegrino, Donald, 2009. "Towards an explanatory and computational theory of scientific discovery," Journal of Informetrics, Elsevier, vol. 3(3), pages 191-209.
    14. Shawn J. Green & Jon Brendsel, 2008. "Key discoveries often originate with lone researchers," Nature, Nature, vol. 456(7220), pages 315-315, November.
    15. Richard Van Noorden & Brendan Maher & Regina Nuzzo, 2014. "The top 100 papers," Nature, Nature, vol. 514(7524), pages 550-553, October.
    16. R Grant Steen & Arturo Casadevall & Ferric C Fang, 2013. "Why Has the Number of Scientific Retractions Increased?," PLOS ONE, Public Library of Science, vol. 8(7), pages 1-9, July.
    17. Anthony F. J. van Raan, 2004. "Sleeping Beauties in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(3), pages 467-472, March.
    18. Ponomarev, Ilya V. & Williams, Duane E. & Hackett, Charles J. & Schnell, Joshua D. & Haak, Laurel L., 2014. "Predicting highly cited papers: A Method for Early Detection of Candidate Breakthroughs," Technological Forecasting and Social Change, Elsevier, vol. 81(C), pages 49-55.
    19. Jos J. Winnink & Robert J. W. Tijssen & Anthony F. J. van Raan, 2016. "Theory‐changing breakthroughs in science: The impact of research teamwork on scientific discoveries," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(5), pages 1210-1223, May.
    20. Susanne E. Baumgartner & Loet Leydesdorff, 2014. "Group-based trajectory modeling (GBTM) of citations in scholarly literature: Dynamic qualities of “transient” and “sticky knowledge claims”," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 797-811, April.
    21. J. J. Winnink & Robert J. W. Tijssen, 2015. "Early stage identification of breakthroughs at the interface of science and technology: lessons drawn from a landmark publication," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 113-134, January.
    22. Jonathan Adams, 2005. "Early citation counts correlate with accumulated impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(3), pages 567-581, June.
    23. Juan D Rogers, 2010. "Citation analysis of nanotechnology at the field level: implications of R&D evaluation," Research Evaluation, Oxford University Press, vol. 19(4), pages 281-290, October.
    24. Marten Scheffer & Jordi Bascompte & William A. Brock & Victor Brovkin & Stephen R. Carpenter & Vasilis Dakos & Hermann Held & Egbert H. van Nes & Max Rietkerk & George Sugihara, 2009. "Early-warning signals for critical transitions," Nature, Nature, vol. 461(7260), pages 53-59, September.
    25. J. J. Winnink & Robert J. W. Tijssen, 2014. "R&D dynamics and scientific breakthroughs in HIV/AIDS drugs development: the case of Integrase Inhibitors," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 1-16, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Libo Sheng & Dongqing Lyu & Xuanmin Ruan & Hongquan Shen & Ying Cheng, 2023. "The association between prior knowledge and the disruption of an article," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4731-4751, August.
    2. Min, Chao & Bu, Yi & Sun, Jianjun, 2021. "Predicting scientific breakthroughs based on knowledge structure variations," Technological Forecasting and Social Change, Elsevier, vol. 164(C).
    3. Xian Li & Ronald Rousseau & Liming Liang & Fangjie Xi & Yushuang Lü & Yifan Yuan & Xiaojun Hu, 2022. "Is low interdisciplinarity of references an unexpected characteristic of Nobel Prize winning research?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 2105-2122, April.
    4. Ma, Yaxue & Ba, Zhichao & Zhao, Haiping & Sun, Jianjun, 2023. "How to configure intellectual capital of research teams for triggering scientific breakthroughs: Exploratory study in the field of gene editing," Journal of Informetrics, Elsevier, vol. 17(4).
    5. Xue Wang & Xuemei Yang & Jian Du & Xuwen Wang & Jiao Li & Xiaoli Tang, 2021. "A deep learning approach for identifying biomedical breakthrough discoveries using context analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5531-5549, July.
    6. Li, Xin & Wen, Yang & Jiang, Jiaojiao & Daim, Tugrul & Huang, Lucheng, 2022. "Identifying potential breakthrough research: A machine learning method using scientific papers and Twitter data," Technological Forecasting and Social Change, Elsevier, vol. 184(C).
    7. Heng Huang & Donghua Zhu & Xuefeng Wang, 2022. "Evaluating scientific impact of publications: combining citation polarity and purpose," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5257-5281, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. J. J. Winnink & Robert J. W. Tijssen, 2015. "Early stage identification of breakthroughs at the interface of science and technology: lessons drawn from a landmark publication," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 113-134, January.
    2. Min, Chao & Bu, Yi & Sun, Jianjun, 2021. "Predicting scientific breakthroughs based on knowledge structure variations," Technological Forecasting and Social Change, Elsevier, vol. 164(C).
    3. Li, Xin & Wen, Yang & Jiang, Jiaojiao & Daim, Tugrul & Huang, Lucheng, 2022. "Identifying potential breakthrough research: A machine learning method using scientific papers and Twitter data," Technological Forecasting and Social Change, Elsevier, vol. 184(C).
    4. Xian Li & Ronald Rousseau & Liming Liang & Fangjie Xi & Yushuang Lü & Yifan Yuan & Xiaojun Hu, 2022. "Is low interdisciplinarity of references an unexpected characteristic of Nobel Prize winning research?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 2105-2122, April.
    5. Krzysztof Klincewicz, 2016. "The emergent dynamics of a technological research topic: the case of graphene," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 319-345, January.
    6. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    7. Li, Xin & Ma, Xiaodi & Feng, Ye, 2024. "Early identification of breakthrough research from sleeping beauties using machine learning," Journal of Informetrics, Elsevier, vol. 18(2).
    8. Fenghua Wang & Ying Fan & An Zeng & Zengru Di, 2019. "Can we predict ESI highly cited publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 109-125, January.
    9. Small, Henry & Boyack, Kevin W. & Klavans, Richard, 2014. "Identifying emerging topics in science and technology," Research Policy, Elsevier, vol. 43(8), pages 1450-1467.
    10. Llopis, Oscar & D'Este, Pablo & McKelvey, Maureen & Yegros, Alfredo, 2022. "Navigating multiple logics: Legitimacy and the quest for societal impact in science," Technovation, Elsevier, vol. 110(C).
    11. Jianhua Hou & Xiucai Yang, 2019. "Patent sleeping beauties: evolutionary trajectories and identification methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(1), pages 187-215, July.
    12. Shiyun Wang & Yaxue Ma & Jin Mao & Yun Bai & Zhentao Liang & Gang Li, 2023. "Quantifying scientific breakthroughs by a novel disruption indicator based on knowledge entities," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(2), pages 150-167, February.
    13. Bornmann, Lutz & Tekles, Alexander & Zhang, Helena H. & Ye, Fred Y., 2019. "Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data," Journal of Informetrics, Elsevier, vol. 13(4).
    14. Jos J. Winnink & Robert J. W. Tijssen & Anthony F. J. van Raan, 2016. "Theory‐changing breakthroughs in science: The impact of research teamwork on scientific discoveries," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(5), pages 1210-1223, May.
    15. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    16. Xue Wang & Xuemei Yang & Jian Du & Xuwen Wang & Jiao Li & Xiaoli Tang, 2021. "A deep learning approach for identifying biomedical breakthrough discoveries using context analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5531-5549, July.
    17. Dejian Yu & Libo Sheng & Shunshun Shi, 2023. "A retrospective analysis of Journal of Forecasting: From 1982 to 2019," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(4), pages 1008-1035, July.
    18. Zhang, Ruizhi & Wang, Jian & Mei, Yajun, 2017. "Search for evergreens in science: A functional data analysis," Journal of Informetrics, Elsevier, vol. 11(3), pages 629-644.
    19. Daniela Filippo & Rafael Aleixandre-Benavent & Elías Sanz-Casado, 2020. "Toward a classification of Spanish scholarly journals in social sciences and humanities considering their impact and visibility," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1709-1732, November.
    20. De Filippo, Daniela & Gorraiz, Juan, 2020. "Is the Emerging Source Citation Index an aid to assess the citation impact in social science and humanities?," Journal of Informetrics, Elsevier, vol. 14(4).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:tefoso:v:146:y:2019:i:c:p:673-686. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.sciencedirect.com/science/journal/00401625 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.