IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v71y2007i2d10.1007_s11192-007-1659-2.html
   My bibliography  Save this article

On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications

Author

Listed:
  • Günter Krampen

    (University of Trier)

  • Ralf Becker

    (University of Trier)

  • Ute Wahner

    (University of Trier)

  • Leo Montada

    (University of Trier)

Abstract

In reference to the increasing significance of citation counting in evaluations of scientists and science institutes as well as in science historiography, it is analyzed empirically what is cited in which frequency and what types of citations in scientific texts are used. Content analyses refer to numbers of references, self-references, publication language of references cited, publication types of references cited, and type of citation within the texts. Validity of citation counting is empirically analyzed with reference to random samples of English and German journal articles as well as German textbooks, encyclopedias, and test-manuals from psychology. Results show that 25% of all citations are perfunctory, more than 50% of references are journal articles and up to 40% are books and book-chapters, 10% are self-references. Differences between publications from various psychological sub-disciplines, publication languages, and types of publication are weak. Thus, validity of evaluative citation counting is limited because at least one quarter refers to perfunctory citations exhibiting a very low information utility level and by the fact that existing citation-databases refer to journal articles only.

Suggested Citation

  • Günter Krampen & Ralf Becker & Ute Wahner & Leo Montada, 2007. "On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 71(2), pages 191-202, May.
    • Handle: RePEc:spr:scient:v:71:y:2007:i:2:d:10.1007_s11192-007-1659-2
    DOI: 10.1007/s11192-007-1659-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-007-1659-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-007-1659-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Terrence A. Brooks, 1986. "Evidence of complex citer motivations," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 37(1), pages 34-36, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chi-Shiou Lin, 2018. "An analysis of citation functions in the humanities and social sciences research from the perspective of problematic citation analysis assumptions," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 797-813, August.
    2. Lutz Bornmann & Hanna Herich & Hanna Joos & Hans-Dieter Daniel, 2012. "In public peer review of submitted manuscripts, how do reviewer comments differ from comments written by interested members of the scientific community? A content analysis of comments written for Atmo," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 915-929, December.
    3. Ponomariov, Branco & Toivanen, Hannes, 2014. "Knowledge flows and bases in emerging economy innovation systems: Brazilian research 2005–2009," Research Policy, Elsevier, vol. 43(3), pages 588-596.
    4. Wu, Jiang & Ou, Guiyan & Liu, Xiaohui & Dong, Ke, 2022. "How does academic education background affect top researchers’ performance? Evidence from the field of artificial intelligence," Journal of Informetrics, Elsevier, vol. 16(2).
    5. Teplitskiy, Misha & Duede, Eamon & Menietti, Michael & Lakhani, Karim R., 2022. "How status of research papers affects the way they are read and cited," Research Policy, Elsevier, vol. 51(4).
    6. Lowe, D. Jordan & Van Fleet, David D., 2009. "Scholarly achievement and accounting journal editorial board membership," Journal of Accounting Education, Elsevier, vol. 27(4), pages 197-209.
    7. Carlo D'Ippoliti, 2021. "“Many‐Citedness”: Citations Measure More Than Just Scientific Quality," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1271-1301, December.
    8. Andrea Diem & Stefan C. Wolter, 2011. "The Use of Bibliometrics to Measure Research Performance in Education Sciences," Economics of Education Working Paper Series 0066, University of Zurich, Department of Business Administration (IBW), revised May 2013.
    9. Stremersch, Stefan & Camacho, Nuno & Vanneste, Sofie & Verniers, Isabel, 2015. "Unraveling scientific impact: Citation types in marketing journals," International Journal of Research in Marketing, Elsevier, vol. 32(1), pages 64-77.
    10. Iñaki Ucar & Felipe López-Fernandino & Pablo Rodriguez-Ulibarri & Laura Sesma-Sanchez & Veronica Urrea-Micó & Joaquín Sevilla, 2014. "Growth in the number of references in engineering journal papers during the 1972–2013 period," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 1855-1864, March.
    11. Maurice Poirrier & Sebastián Moreno & Gonzalo Huerta-Cánepa, 2021. "Robust h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 1969-1981, March.
    12. Jessica Petersen & Fabian Hattke & Rick Vogel, 2017. "Editorial governance and journal impact: a study of management and business journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1593-1614, September.
    13. Masaki Eto, 2013. "Evaluations of context-based co-citation searching," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 651-673, February.
    14. Stremersch, S. & Camacho, N.M.A. & Vanneste, S. & Verniers, I.W.J., 2014. "Unraveling Scientific Impact," ERIM Report Series Research in Management ERS-2014-014-MKT, Erasmus Research Institute of Management (ERIM), ERIM is the joint research institute of the Rotterdam School of Management, Erasmus University and the Erasmus School of Economics (ESE) at Erasmus University Rotterdam.
    15. Günter Krampen, 2010. "Acceleration of citing behavior after the millennium? Exemplary bibliometric reference analyses for psychology journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(2), pages 507-513, May.
    16. Kousha, Kayvan & Thelwall, Mike, 2018. "Can Microsoft Academic help to assess the citation impact of academic books?," Journal of Informetrics, Elsevier, vol. 12(3), pages 972-984.
    17. Huang, Chen-Hao & Liu, John S. & Ho, Mei Hsiu-Ching & Chou, Tzu-Chuan, 2022. "Towards more convergent main paths: A relevance-based approach," Journal of Informetrics, Elsevier, vol. 16(3).
    18. Zhang, Chengzhi & Zhou, Qingqing, 2020. "Assessing books’ depth and breadth via multi-level mining on tables of contents," Journal of Informetrics, Elsevier, vol. 14(2).
    19. José Álvarez-García & Amador Durán-Sánchez & Néstor Montalván-Burbano & María de la Cruz del Río-Rama, 2023. "Spanish Journal of Finance and Accounting (SJFA): Mapping of Knowledge over the Last 25 Years," Publications, MDPI, vol. 11(1), pages 1-19, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dangzhi Zhao & Andreas Strotmann, 2020. "Telescopic and panoramic views of library and information science research 2011–2018: a comparison of four weighting schemes for author co-citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 255-270, July.
    2. Nigel Harwood, 2008. "Publication outlets and their effect on academic writers’ citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 77(2), pages 253-265, November.
    3. Mike Thelwall, 2012. "Journal impact evaluation: a webometric perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 429-441, August.
    4. Dongqing Lyu & Xuanmin Ruan & Juan Xie & Ying Cheng, 2021. "The classification of citing motivations: a meta-synthesis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3243-3264, April.
    5. Linhong Xu & Kun Ding & Yuan Lin, 2022. "Do negative citations reduce the impact of cited papers?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 1161-1186, February.
    6. Xie, Zheng & Ouyang, Zhenzheng & Liu, Qi & Li, Jianping, 2016. "A geometric graph model for citation networks of exponentially growing scientific papers," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 456(C), pages 167-175.
    7. Adrián Kovács & Bart Looy & Bruno Cassiman, 2015. "Exploring the scope of open innovation: a bibliometric review of a decade of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 951-983, September.
    8. Reut Snir & Gilad Ravid, 2016. "Global nanotechnology regulatory governance from a network analysis perspective," Regulation & Governance, John Wiley & Sons, vol. 10(4), pages 314-334, December.
    9. Clough, James R. & Evans, Tim S., 2016. "What is the dimension of citation space?," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 448(C), pages 235-247.
    10. Mao, Jin & Liang, Zhentao & Cao, Yujie & Li, Gang, 2020. "Quantifying cross-disciplinary knowledge flow from the perspective of content: Introducing an approach based on knowledge memes," Journal of Informetrics, Elsevier, vol. 14(4).
    11. Binglu Wang & Yi Bu & Yang Xu, 2018. "A quantitative exploration on reasons for citing articles from the perspective of cited authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 675-687, August.
    12. Hou, Jianhua & Wang, Dongyi & Li, Jing, 2022. "A new method for measuring the originality of academic articles based on knowledge units in semantic networks," Journal of Informetrics, Elsevier, vol. 16(3).
    13. Tanzila Ahmed & Ben Johnson & Charles Oppenheim & Catherine Peck, 2004. "Highly cited old papers and the reasons why they continue to be cited. Part II., The 1953 Watson and Crick article on the structure of DNA," Scientometrics, Springer;Akadémiai Kiadó, vol. 61(2), pages 147-156, October.
    14. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    15. Bikun Chen & Dannan Deng & Zhouyan Zhong & Chengzhi Zhang, 2020. "Exploring linguistic characteristics of highly browsed and downloaded academic articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1769-1790, March.
    16. P. Vinkler, 2010. "Indicators are the essence of scientometrics and bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 861-866, December.
    17. Hu, Zhigang & Chen, Chaomei & Liu, Zeyuan, 2013. "Where are citations located in the body of scientific articles? A study of the distributions of citation locations," Journal of Informetrics, Elsevier, vol. 7(4), pages 887-896.
    18. Lina Zhou & Uchechukwuka Amadi & Dongsong Zhang, 2020. "Is Self-Citation Biased? An Investigation via the Lens of Citation Polarity, Density, and Location," Information Systems Frontiers, Springer, vol. 22(1), pages 77-90, February.
    19. Elizabeth S. Vieira & José A. N. F. Gomes, 2011. "An impact indicator for researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(2), pages 607-629, November.
    20. Amalia Mas-Bleda & Mike Thelwall & Kayvan Kousha & Isidro F. Aguillo, 2014. "Do highly cited researchers successfully use the social web?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 337-356, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:71:y:2007:i:2:d:10.1007_s11192-007-1659-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.