IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v4y2010i1p124-135.html
   My bibliography  Save this article

Using the Web for research evaluation: The Integrated Online Impact indicator

Author

Listed:
  • Kousha, Kayvan
  • Thelwall, Mike
  • Rezaie, Somayeh

Abstract

Previous research has shown that citation data from different types of Web sources can potentially be used for research evaluation. Here we introduce a new combined Integrated Online Impact (IOI) indicator. For a case study, we selected research articles published in the Journal of the American Society for Information Science & Technology (JASIST) and Scientometrics in 2003. We compared the citation counts from Web of Science (WoS) and Scopus with five online sources of citation data including Google Scholar, Google Books, Google Blogs, PowerPoint presentations and course reading lists. The mean and median IOI was nearly twice as high as both WoS and Scopus, confirming that online citations are sufficiently numerous to be useful for the impact assessment of research. We also found significant correlations between conventional and online impact indicators, confirming that both assess something similar in scholarly communication. Further analysis showed that the overall percentage for unique Google Scholar citations outside the WoS were 73% and 60% for the articles published in JASIST and Scientometrics, respectively. An important conclusion is that in subject areas where wider types of intellectual impact indicators outside the WoS and Scopus databases are needed for research evaluation, IOI can be used to help monitor research performance.

Suggested Citation

  • Kousha, Kayvan & Thelwall, Mike & Rezaie, Somayeh, 2010. "Using the Web for research evaluation: The Integrated Online Impact indicator," Journal of Informetrics, Elsevier, vol. 4(1), pages 124-135.
    • Handle: RePEc:eee:infome:v:4:y:2010:i:1:p:124-135
    DOI: 10.1016/j.joi.2009.10.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157709000777
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2009.10.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bornmann, Lutz & Marx, Werner & Schier, Hermann & Rahm, Erhard & Thor, Andreas & Daniel, Hans-Dieter, 2009. "Convergent validity of bibliometric Google Scholar data in the field of chemistry—Citation counts for papers that were accepted by Angewandte Chemie International Edition or rejected but published els," Journal of Informetrics, Elsevier, vol. 3(1), pages 27-35.
    2. Liwen Vaughan & Debora Shaw, 2005. "Web citation data for impact assessment: A comparison of four science disciplines," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 56(10), pages 1075-1087, August.
    3. Kayvan Kousha & Mike Thelwall, 2008. "Assessing the impact of disciplinary research on teaching: An automatic analysis of online syllabuses," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(13), pages 2060-2069, November.
    4. Norris, Michael & Oppenheim, Charles, 2007. "Comparing alternatives to the Web of Science for coverage of the social sciences’ literature," Journal of Informetrics, Elsevier, vol. 1(2), pages 161-169.
    5. Lokman I. Meho & Kiduk Yang, 2007. "Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(13), pages 2105-2125, November.
    6. Anton J. Nederhof, 2006. "Bibliometric monitoring of research performance in the Social Sciences and the Humanities: A Review," Scientometrics, Springer;Akadémiai Kiadó, vol. 66(1), pages 81-100, January.
    7. Judit Bar-Ilan, 2008. "Which h-index? — A comparison of WoS, Scopus and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 257-271, February.
    8. Anne‐Wil Harzing & Ron van der Wal, 2009. "A Google Scholar h‐index for journals: An alternative metric to measure journal impact in economics and business," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 41-46, January.
    9. Judit Bar-Ilan, 2004. "A microscopic link analysis of academic institutions within a country — the case of Israel," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(3), pages 391-403, March.
    10. Liwen Vaughan & Debora Shaw, 2003. "Bibliographic and Web citations: What is the difference?," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 54(14), pages 1313-1322, December.
    11. Kayvan Kousha & Mike Thelwall, 2009. "Google book search: Citation analysis for social science and the humanities," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(8), pages 1537-1549, August.
    12. Kayvan Kousha & Mike Thelwall, 2008. "Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 273-294, February.
    13. Mike Thelwall & Kayvan Kousha, 2008. "Online presentations as a source of scientific impact? An analysis of PowerPoint files citing academic journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(5), pages 805-815, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pardeep Sud & Mike Thelwall, 2014. "Evaluating altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1131-1143, February.
    2. Mike Thelwall, 2012. "Journal impact evaluation: a webometric perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 429-441, August.
    3. Qingqing Zhou & Chengzhi Zhang & Star X. Zhao & Bikun Chen, 2016. "Measuring book impact based on the multi-granularity online review mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1435-1455, June.
    4. George A. Barnett & Han Woo Park, 2023. "Co-authorship among the Fellows of the International Communication Association," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3401-3418, June.
    5. Xuemei Li & Mike Thelwall & Dean Giustini, 2012. "Validating online reference managers for scholarly impact measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 461-471, May.
    6. Amalia Más-Bleda & Isidro F. Aguillo, 2013. "Can a personal website be useful as an information source to assess individual scientists? The case of European highly cited researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 51-67, July.
    7. Amalia Mas-Bleda & Mike Thelwall & Kayvan Kousha & Isidro F. Aguillo, 2014. "Do highly cited researchers successfully use the social web?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 337-356, October.
    8. Xianwen Wang & Wenli Mao & Shenmeng Xu & Chunbo Zhang, 2014. "Usage history of scientific literature: Nature metrics and metrics of Nature publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 1923-1933, March.
    9. Enrique Orduña-Malea, 2021. "Dot-science top level domain: Academic websites or dumpsites?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3565-3591, April.
    10. Geuens, Maggie, 2011. "Where does business research go from here? Food-for-thought on academic papers in business research," Journal of Business Research, Elsevier, vol. 64(10), pages 1104-1107, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Mike Thelwall, 2012. "Journal impact evaluation: a webometric perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 429-441, August.
    3. Halevi, Gali & Moed, Henk & Bar-Ilan, Judit, 2017. "Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the Literature," Journal of Informetrics, Elsevier, vol. 11(3), pages 823-834.
    4. Xuemei Li & Mike Thelwall & Dean Giustini, 2012. "Validating online reference managers for scholarly impact measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 461-471, May.
    5. Amalia Mas-Bleda & Mike Thelwall, 2016. "Can alternative indicators overcome language biases in citation counts? A comparison of Spanish and UK research," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2007-2030, December.
    6. Massimo Franceschet, 2010. "A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(1), pages 243-258, April.
    7. Amalia Mas-Bleda & Mike Thelwall & Kayvan Kousha & Isidro F. Aguillo, 2014. "Do highly cited researchers successfully use the social web?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 337-356, October.
    8. Pardeep Sud & Mike Thelwall, 2014. "Evaluating altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1131-1143, February.
    9. Anne-Wil Harzing, 2013. "A preliminary test of Google Scholar as a source for citation data: a longitudinal study of Nobel prize winners," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1057-1075, March.
    10. Judit Bar-Ilan, 2010. "Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(3), pages 495-506, March.
    11. Alberto Martín-Martín & Enrique Orduna-Malea & Emilio Delgado López-Cózar, 2018. "Coverage of highly-cited documents in Google Scholar, Web of Science, and Scopus: a multidisciplinary comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2175-2188, September.
    12. Andrea Giovanni Nuzzolese & Paolo Ciancarini & Aldo Gangemi & Silvio Peroni & Francesco Poggi & Valentina Presutti, 2019. "Do altmetrics work for assessing research quality?," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 539-562, February.
    13. Kuang-hua Chen & Muh-chyun Tang & Chun-mei Wang & Jieh Hsiang, 2015. "Exploring alternative metrics of scholarly performance in the social sciences and humanities in Taiwan," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 97-112, January.
    14. Moed, Henk F. & Bar-Ilan, Judit & Halevi, Gali, 2016. "A new methodology for comparing Google Scholar and Scopus," Journal of Informetrics, Elsevier, vol. 10(2), pages 533-551.
    15. García-Pérez, Miguel A., 2011. "Strange attractors in the Web of Science database," Journal of Informetrics, Elsevier, vol. 5(1), pages 214-218.
    16. Martin-Martin, Alberto & Orduna-Malea, Enrique & Harzing, Anne-Wil & Delgado López-Cózar, Emilio, 2017. "Can we use Google Scholar to identify highly-cited documents?," Journal of Informetrics, Elsevier, vol. 11(1), pages 152-163.
    17. Nabil Amara & Réjean Landry, 2012. "Counting citations in the field of business and management: why use Google Scholar rather than the Web of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 553-581, December.
    18. Sergio Copiello, 2019. "The open access citation premium may depend on the openness and inclusiveness of the indexing database, but the relationship is controversial because it is ambiguous where the open access boundary lie," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 995-1018, November.
    19. Peder Olesen Larsen & Markus Ins, 2010. "The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 575-603, September.
    20. García-Pérez, Miguel A., 2012. "An extension of the h index that covers the tail and the top of the citation curve and allows ranking researchers with similar h," Journal of Informetrics, Elsevier, vol. 6(4), pages 689-699.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:4:y:2010:i:1:p:124-135. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.