IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v5y2011i4p629-648.html
   My bibliography  Save this article

Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence

Author

Listed:
  • Serenko, Alexander
  • Dohan, Michael

Abstract

The purpose of this study is to: (1) develop a ranking of peer-reviewed AI journals; (2) compare the consistency of journal rankings developed with two dominant ranking techniques, expert surveys and journal impact measures; and (3) investigate the consistency of journal ranking scores assigned by different categories of expert judges. The ranking was constructed based on the survey of 873 active AI researchers who ranked the overall quality of 182 peer-reviewed AI journals. It is concluded that expert surveys and citation impact journal ranking methods cannot be used as substitutes. Instead, they should be used as complementary approaches. The key problem of the expert survey ranking technique is that in their ranking decisions, respondents are strongly influenced by their current research interests. As a result, their scores merely reflect their present research preferences rather than an objective assessment of each journal's quality. In addition, the application of the expert survey method favors journals that publish more articles per year.

Suggested Citation

  • Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    • Handle: RePEc:eee:infome:v:5:y:2011:i:4:p:629-648
    DOI: 10.1016/j.joi.2011.06.002
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157711000666
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2011.06.002?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Leo Egghe, 2006. "Theory and practise of the g-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 131-152, October.
    2. Chiang Kao & Hsiou-Wei Lin & San-Lin Chung & Wei-Chi Tsai & Jyh-Shen Chiou & Yen-Liang Chen & Liang-Hsuan Chen & Shih-Chieh Fang & Hwei-Lan Pao, 2008. "Ranking Taiwanese management journals: A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 95-115, July.
    3. Gunther Maier, 2006. "Impact factors and peer judgment: The case of regional science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 651-667, December.
    4. Christian Schloegl & Wolfgang G. Stock, 2004. "Impact and relevance of LIS journals: A scientometric analysis of international and German‐language LIS journals—Citation analysis versus reader survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 55(13), pages 1155-1168, November.
    5. Michael D. Gordon, 1982. "Citation ranking versus subjective evaluation in the determination of journal hierachies in the social sciences," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 33(1), pages 55-57, January.
    6. Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2007. "Generalized Hirsch h-index for disclosing latent facts in citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(2), pages 253-280, August.
    7. Lokman I. Meho & Kiduk Yang, 2007. "Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(13), pages 2105-2125, November.
    8. Forgionne, Guisseppi A. & Kohli, Rajiv & Jennings, Darniet, 2002. "An AHP analysis of quality in AI and DSS journals," Omega, Elsevier, vol. 30(3), pages 171-183, June.
    9. Anthony J. Nederhof & Marc Luwel & Henk F. Moed, 2001. "Assessing the quality of scholarly journals in Linguistics:An alternative to citation-based journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(1), pages 241-265, April.
    10. Frandsen, Tove Faber, 2007. "Journal self-citations—Analysing the JIF mechanism," Journal of Informetrics, Elsevier, vol. 1(1), pages 47-58.
    11. Serenko, Alexander, 2010. "The development of an AI journal ranking based on the revealed preference approach," Journal of Informetrics, Elsevier, vol. 4(4), pages 447-459.
    12. Alexander Serenko & Nick Bontis, 2009. "A citation-based ranking of the business ethics scholarly journals," International Journal of Business Governance and Ethics, Inderscience Enterprises Ltd, vol. 4(4), pages 390-399.
    13. Mark R. Elkins & Christopher G. Maher & Robert D. Herbert & Anne M. Moseley & Catherine Sherrington, 2010. "Correlation between the Journal Impact Factor and three other journal citation indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 81-93, October.
    14. Serenko, Alexander & Bontis, Nick, 2011. "What's familiar is excellent: The impact of exposure effect on perceived journal quality," Journal of Informetrics, Elsevier, vol. 5(1), pages 219-223.
    15. Josephine E. Olson, 2005. "Top-25-Business-School Professors Rate Journals in Operations Management and Related Fields," Interfaces, INFORMS, vol. 35(4), pages 323-338, August.
    16. R. Rousseau, 1999. "Temporal differences in self-citation rates of scientific journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 44(3), pages 521-531, March.
    17. Frank L DuBois & David Reeb, 2000. "Ranking the International Business Journals," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 31(4), pages 689-704, December.
    18. Jonathan C. Catling & Victoria L. Mason & Dominic Upton, 2009. "Quality is in the eye of the beholder? An evaluation of impact factors and perception of journal prestige in the UK," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 333-345, November.
    19. Donohue, Joan M. & Fox, Jeremy B., 2000. "A multi-method evaluation of journals in the decision and management sciences by US academics," Omega, Elsevier, vol. 28(1), pages 17-36, February.
    20. Sandra Rousseau, 2008. "Journal evaluation by environmental and resource economists: A survey," Scientometrics, Springer;Akadémiai Kiadó, vol. 77(2), pages 223-233, November.
    21. Vanclay, Jerome K., 2008. "Ranking forestry journals using the h-index," Journal of Informetrics, Elsevier, vol. 2(4), pages 326-334.
    22. Vikas Mittal & Lawrence Feick & Feisal Murshed, 2008. "Publish and Prosper: The Financial Impact of Publishing by Marketing Faculty," Marketing Science, INFORMS, vol. 27(3), pages 430-442, 05-06.
    23. William H. Starbuck, 2005. "How Much Better Are the Most-Prestigious Journals? The Statistics of Academic Publication," Organization Science, INFORMS, vol. 16(2), pages 180-200, April.
    24. Per O. Seglen, 1992. "The skewness of science," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 43(9), pages 628-638, October.
    25. Franceschet, Massimo, 2010. "Journal influence factors," Journal of Informetrics, Elsevier, vol. 4(3), pages 239-248.
    26. Michael C. Calver & J. Stuart Bradley, 2009. "Should we use the mean citations per paper to summarise a journal’s impact or to rank journals in the same field?," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(3), pages 611-615, December.
    27. Paul R. McAllister & Richard C. Anderson & Francis Narin, 1980. "Comparison of peer and citation assessment of the influence of scientific journals," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 31(3), pages 147-152, May.
    28. Linda Butler, 2002. "Identifying "highly-rated" journals - an Australian case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 207-227, February.
    29. Gaby Haddow & Paul Genoni, 2010. "Citation analysis and peer ranking of Australian social science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 471-487, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    2. Tânia F. G. G. Cova & Alberto A. C. C. Pais & Sebastião J. Formosinho, 2013. "Iberian universities: a characterisation from ESI rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1239-1251, March.
    3. Domingo Docampo & Vicente Safón, 2021. "Journal ratings: a paper affiliation methodology," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8063-8090, September.
    4. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    5. Antonia Ferrer-Sapena & Susana Díaz-Novillo & Enrique A. Sánchez-Pérez, 2017. "Measuring Time-Dynamics and Time-Stability of Journal Rankings in Mathematics and Physics by Means of Fractional p -Variations," Publications, MDPI, vol. 5(3), pages 1-14, September.
    6. Emanuel Kulczycki & Ewa A. Rozkosz, 2017. "Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 417-442, April.
    7. Johan Lyhagen & Per Ahlgren, 2020. "Uncertainty and the ranking of economics journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2545-2560, December.
    8. Saarela, Mirka & Kärkkäinen, Tommi & Lahtonen, Tommi & Rossi, Tuomo, 2016. "Expert-based versus citation-based ranking of scholarly and scientific publication channels," Journal of Informetrics, Elsevier, vol. 10(3), pages 693-718.
    9. Thelwall, Mike & Fairclough, Ruth, 2015. "Geometric journal impact factors correcting for individual highly cited articles," Journal of Informetrics, Elsevier, vol. 9(2), pages 263-272.
    10. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2013. "National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 311-324, April.
    11. Lior Rokach, 2012. "Applying the Publication Power Approach to Artificial Intelligence Journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(6), pages 1270-1277, June.
    12. Serenko, Alexander & Bontis, Nick, 2013. "First in, best dressed: The presence of order-effect bias in journal ranking surveys," Journal of Informetrics, Elsevier, vol. 7(1), pages 138-144.
    13. Currie, Russell R. & Pandher, Gurupdesh S., 2020. "Finance journal rankings: Active scholar assessment revisited," Journal of Banking & Finance, Elsevier, vol. 111(C).
    14. Jerome K. Vanclay & Lutz Bornmann, 2012. "Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 751-771, June.
    15. Ahlgren, Per & Waltman, Ludo, 2014. "The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments," Journal of Informetrics, Elsevier, vol. 8(4), pages 985-996.
    16. Laura Vana & Ronald Hochreiter & Kurt Hornik, 2016. "Computing a journal meta-ranking using paired comparisons and adaptive lasso estimators," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 229-251, January.
    17. Haddawy, Peter & Hassan, Saeed-Ul & Asghar, Awais & Amin, Sarah, 2016. "A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality," Journal of Informetrics, Elsevier, vol. 10(1), pages 162-173.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    2. Lior Rokach, 2012. "Applying the Publication Power Approach to Artificial Intelligence Journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(6), pages 1270-1277, June.
    3. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    4. Serenko, Alexander, 2010. "The development of an AI journal ranking based on the revealed preference approach," Journal of Informetrics, Elsevier, vol. 4(4), pages 447-459.
    5. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    6. Emanuel Kulczycki & Ewa A. Rozkosz, 2017. "Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 417-442, April.
    7. Kam C. Chan & Anna Fung & Hung-Gay Fung & Jot Yau, 2016. "A Citation Analysis of Business Ethics Research: A Global Perspective," Journal of Business Ethics, Springer, vol. 136(3), pages 557-573, July.
    8. Fiorenzo Franceschini & Domenico Maisano & Anna Perotti & Andrea Proto, 2010. "Analysis of the ch-index: an indicator to evaluate the diffusion of scientific research output by citers," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 203-217, October.
    9. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    10. Yi-Ching Liaw & Te-Yi Chan & Chin-Yuan Fan & Cheng-Hsin Chiang, 2014. "Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 17-37, October.
    11. Christoph Bartneck & Servaas Kokkelmans, 2011. "Detecting h-index manipulation through self-citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(1), pages 85-98, April.
    12. Petridis, Konstantinos & Malesios, Chrisovalantis & Arabatzis, Garyfallos & Thanassoulis, Emmanuel, 2013. "Efficiency analysis of forestry journals: Suggestions for improving journals’ quality," Journal of Informetrics, Elsevier, vol. 7(2), pages 505-521.
    13. Holsapple, Clyde W. & Lee-Post, Anita, 2010. "Behavior-based analysis of knowledge dissemination channels in operations management," Omega, Elsevier, vol. 38(3-4), pages 167-178, June.
    14. Chiang Kao & Hsiou-Wei Lin & San-Lin Chung & Wei-Chi Tsai & Jyh-Shen Chiou & Yen-Liang Chen & Liang-Hsuan Chen & Shih-Chieh Fang & Hwei-Lan Pao, 2008. "Ranking Taiwanese management journals: A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 95-115, July.
    15. Franceschini, Fiorenzo & Maisano, Domenico, 2010. "The Hirsch spectrum: A novel tool for analyzing scientific journals," Journal of Informetrics, Elsevier, vol. 4(1), pages 64-73.
    16. Thelwall, Mike & Fairclough, Ruth, 2015. "Geometric journal impact factors correcting for individual highly cited articles," Journal of Informetrics, Elsevier, vol. 9(2), pages 263-272.
    17. J Mingers, 2009. "Measuring the research contribution of management academics using the Hirsch-index," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 60(9), pages 1143-1153, September.
    18. Andrea Diem & Stefan C. Wolter, 2011. "The Use of Bibliometrics to Measure Research Performance in Education Sciences," Economics of Education Working Paper Series 0066, University of Zurich, Department of Business Administration (IBW), revised May 2013.
    19. Jerome K. Vanclay & Lutz Bornmann, 2012. "Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 751-771, June.
    20. Zhang, Lin & Thijs, Bart & Glänzel, Wolfgang, 2011. "The diffusion of H-related literature," Journal of Informetrics, Elsevier, vol. 5(4), pages 583-593.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:5:y:2011:i:4:p:629-648. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.