IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v117y2018i3d10.1007_s11192-018-2907-3.html
   My bibliography  Save this article

Towards understanding the relation between citations and research quality in software engineering studies

Author

Listed:
  • Jefferson Seide Molléri

    (BTH - Blekinge Tekniska Högskola)

  • Kai Petersen

    (BTH - Blekinge Tekniska Högskola)

  • Emilia Mendes

    (BTH - Blekinge Tekniska Högskola)

Abstract

The importance of achieving high quality in research practice has been highlighted in different disciplines. At the same time, citations are utilized to measure the impact of academic researchers and institutions. One open question is whether the quality in the reporting of research is related to scientific impact, which would be desired. In this exploratory study we aim to: (1) Investigate how consistently a scoring rubric for rigor and relevance has been used to assess research quality of software engineering studies; (2) Explore the relationship between rigor, relevance and citation count. Through backward snowball sampling we identified 718 primary studies assessed through the scoring rubric. We utilized cluster analysis and conditional inference tree to explore the relationship between quality in the reporting of research (represented by rigor and relevance) and scientiometrics (represented by normalized citations). The results show that only rigor is related to studies’ normalized citations. Besides that, confounding factors are likely to influence the number of citations. The results also suggest that the scoring rubric is not applied the same way by all studies, and one of the likely reasons is because it was found to be too abstract and in need to be further refined. Our findings could be used as a basis to further understand the relation between the quality in the reporting of research and scientific impact, and foster new discussions on how to fairly acknowledge studies for performing well with respect to the emphasized research quality. Furthermore, we highlighted the need to further improve the scoring rubric.

Suggested Citation

  • Jefferson Seide Molléri & Kai Petersen & Emilia Mendes, 2018. "Towards understanding the relation between citations and research quality in software engineering studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 1453-1478, December.
    • Handle: RePEc:spr:scient:v:117:y:2018:i:3:d:10.1007_s11192-018-2907-3
    DOI: 10.1007/s11192-018-2907-3
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-018-2907-3
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-018-2907-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Vahid Garousi & João M. Fernandes, 2017. "Quantity versus impact of software engineering papers: a quantitative study," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(2), pages 963-1006, August.
    2. Dag W. Aksnes, 2006. "Citation rates and perceptions of scientific contribution," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(2), pages 169-185, January.
    3. João M. Fernandes, 2014. "Authorship trends in software engineering," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 257-271, October.
    4. Vahid Garousi, 2015. "A bibliometric analysis of the Turkish software engineering research community," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(1), pages 23-49, October.
    5. Mårtensson, Pär & Fors, Uno & Wallin, Sven-Bertil & Zander, Udo & Nilsson, Gunnar H, 2016. "Evaluating research: A multidisciplinary approach to assessing research practice and quality," Research Policy, Elsevier, vol. 45(3), pages 593-603.
    6. Claes Wohlin, 2009. "A new index for the citation curve of researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 521-533, November.
    7. Mingyang Wang & Guang Yu & Daren Yu, 2011. "Mining typical features for highly cited papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 695-706, June.
    8. Moshe Yitzhaki & Gloria Hammershlag, 2004. "Accessibility and use of information sources among computer scientists and software engineers in Israel: Academy versus industry," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 55(9), pages 832-842, July.
    9. Dag W Aksnes, 2003. "Characteristics of highly cited papers," Research Evaluation, Oxford University Press, vol. 12(3), pages 159-170, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kai Petersen & Nauman Bin Ali, 2021. "An analysis of top author citations in software engineering and a comparison with other fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9147-9183, November.
    2. Alexander Schniedermann, 2021. "A comparison of systematic reviews and guideline-based systematic reviews in medical studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9829-9846, December.
    3. Fernandez Martinez, Roberto & Lostado Lorza, Ruben & Santos Delgado, Ana Alexandra & Piedra, Nelson, 2021. "Use of classification trees and rule-based models to optimize the funding assignment to research projects: A case study of UTPL," Journal of Informetrics, Elsevier, vol. 15(1).
    4. Adilson Vital & Diego R. Amancio, 2022. "A comparative analysis of local similarity metrics and machine learning approaches: application to link prediction in author citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(10), pages 6011-6028, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Vahid Garousi & João M. Fernandes, 2017. "Quantity versus impact of software engineering papers: a quantitative study," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(2), pages 963-1006, August.
    2. Aksnes, Dag W. & Rip, Arie, 2009. "Researchers' perceptions of citations," Research Policy, Elsevier, vol. 38(6), pages 895-905, July.
    3. Tian Yu & Guang Yu & Peng-Yu Li & Liang Wang, 2014. "Citation impact prediction for scientific papers using stepwise regression analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1233-1252, November.
    4. Kenneth Zahringer & Christos Kolympiris & Nicholas Kalaitzandonakes, 2017. "Academic knowledge quality differentials and the quality of firm innovation," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 26(5), pages 821-844.
    5. Waleed M. Sweileh & Sa’ed H. Zyoud & Suleiman Al-Khalil & Samah W. Al-Jabi & Ansam F. Sawalha, 2014. "Assessing the Scientific Research Productivity of the Palestinian Higher Education Institutions," SAGE Open, , vol. 4(3), pages 21582440145, July.
    6. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    7. Aashish Mehta & Patrick Herron & Yasuyuki Motoyama & Richard Appelbaum & Timothy Lenoir, 2012. "Globalization and de-globalization in nanotechnology research: the role of China," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(2), pages 439-458, November.
    8. Mingyang Wang & Guang Yu & Shuang An & Daren Yu, 2012. "Discovery of factors influencing citation impact based on a soft fuzzy rough set model," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 635-644, December.
    9. Wang, Mingyang & Yu, Guang & Xu, Jianzhong & He, Huixin & Yu, Daren & An, Shuang, 2012. "Development a case-based classifier for predicting highly cited papers," Journal of Informetrics, Elsevier, vol. 6(4), pages 586-599.
    10. Ruan, Xuanmin & Zhu, Yuanyang & Li, Jiang & Cheng, Ying, 2020. "Predicting the citation counts of individual papers via a BP neural network," Journal of Informetrics, Elsevier, vol. 14(3).
    11. Yang, Jinqing & Liu, Zhifeng, 2022. "The effect of citation behaviour on knowledge diffusion and intellectual structure," Journal of Informetrics, Elsevier, vol. 16(1).
    12. Li, Xin & Ma, Xiaodi & Feng, Ye, 2024. "Early identification of breakthrough research from sleeping beauties using machine learning," Journal of Informetrics, Elsevier, vol. 18(2).
    13. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    14. Miranda, Ruben & Garcia-Carpintero, Esther, 2018. "Overcitation and overrepresentation of review papers in the most cited papers," Journal of Informetrics, Elsevier, vol. 12(4), pages 1015-1030.
    15. Kai Petersen & Nauman Bin Ali, 2021. "An analysis of top author citations in software engineering and a comparison with other fields," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9147-9183, November.
    16. André Andrian Padial & João Carlos Nabout & Tadeu Siqueira & Luis Mauricio Bini & José Alexandre Felizola Diniz-Filho, 2010. "Weak evidence for determinants of citation frequency in ecological articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 1-12, October.
    17. Mingyang Wang & Shi Li & Guangsheng Chen, 2017. "Detecting latent referential articles based on their vitality performance in the latest 2 years," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1557-1571, September.
    18. Yifan Qian & Wenge Rong & Nan Jiang & Jie Tang & Zhang Xiong, 2017. "Citation regression analysis of computer science publications in different ranking categories and subfields," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1351-1374, March.
    19. Bin Wang & Feng Wu & Lukui Shi, 2023. "AGSTA-NET: adaptive graph spatiotemporal attention network for citation count prediction," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 511-541, January.
    20. Xu, Ran & Baghaei Lakeh, Arash & Ghaffarzadegan, Navid, 2021. "Examining the characteristics of impactful research topics: A case of three decades of HIV-AIDS research," Journal of Informetrics, Elsevier, vol. 15(1).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:117:y:2018:i:3:d:10.1007_s11192-018-2907-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.