IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v111y2017i1d10.1007_s11192-017-2261-x.html
   My bibliography  Save this article

Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland

Author

Listed:
  • Emanuel Kulczycki

    (Adam Mickiewicz University in Poznań)

  • Ewa A. Rozkosz

    (University of Lower Silesia)

Abstract

This article discusses the Polish Journal Ranking, which is used in the research evaluation system in Poland. In 2015, the ranking, which represents all disciplines, allocated 17,437 journals into three lists: A, B, and C. The B list constitutes a ranking of Polish journals that are indexed neither in the Web of Science nor the European Reference Index for the Humanities. This ranking was built by evaluating journals in three dimensions: formal, bibliometric, and expert-based. We have analysed data on 2035 Polish journals from the B list. Our study aims to determine how an expert-based evaluation influenced the results of final evaluation. In our study, we used structural equation modelling, which is regression based, and we designed three pairs of theoretical models for three fields of science: (1) humanities, (2) social sciences, and (3) engineering, natural sciences, and medical sciences. Each pair consisted of the full model and the reduced model (i.e., the model without the expert-based evaluation). Our analysis revealed that the multidimensional evaluation of local journals should not rely only on the bibliometric indicators, which are based on the Web of Science or Scopus. Moreover, we have shown that the expert-based evaluation plays a major role in all fields of science. We conclude with recommendations that the formal evaluation should be reduced to verifiable parameters and that the expert-based evaluation should be based on common guidelines for the experts.

Suggested Citation

  • Emanuel Kulczycki & Ewa A. Rozkosz, 2017. "Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(1), pages 417-442, April.
    • Handle: RePEc:spr:scient:v:111:y:2017:i:1:d:10.1007_s11192-017-2261-x
    DOI: 10.1007/s11192-017-2261-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-017-2261-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-017-2261-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Gunnar Sivertsen, 2016. "Patterns of internationalization and criteria for research assessment in the social sciences and humanities," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 357-368, May.
    2. Gunnar Sivertsen & Birger Larsen, 2012. "Comprehensive bibliographic coverage of the social sciences and humanities in a citation index: an empirical analysis of the potential," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 567-575, May.
    3. Chiang Kao & Hsiou-Wei Lin & San-Lin Chung & Wei-Chi Tsai & Jyh-Shen Chiou & Yen-Liang Chen & Liang-Hsuan Chen & Shih-Chieh Fang & Hwei-Lan Pao, 2008. "Ranking Taiwanese management journals: A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 95-115, July.
    4. Gunther Maier, 2006. "Impact factors and peer judgment: The case of regional science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 651-667, December.
    5. Christian Schloegl & Wolfgang G. Stock, 2004. "Impact and relevance of LIS journals: A scientometric analysis of international and German‐language LIS journals—Citation analysis versus reader survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 55(13), pages 1155-1168, November.
    6. Elizabeth S. Vieira & José A.S. Cabral & José A.N.F. Gomes, 2014. "Definition of a model based on bibliometric indicators for assessing applicants to academic positions," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(3), pages 560-577, March.
    7. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    8. Gualberto Buela-Casal & Pandelis Perakakis & Michael Taylor & Purificación Checa, 2006. "Measuring internationality: Reflections and perspectives on academic journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(1), pages 45-65, April.
    9. Vanclay, Jerome K., 2008. "Ranking forestry journals using the h-index," Journal of Informetrics, Elsevier, vol. 2(4), pages 326-334.
    10. Soledad Moya & Diego Prior & Gonzalo Rodríguez-Pírez, 2015. "Performance-based Incentives and the Behavior of Accounting Academics: Responding to Changes," Accounting Education, Taylor & Francis Journals, vol. 24(3), pages 208-232, June.
    11. Wolfgang Glänzel & Cornelius de Lange, 2002. "A distributional approach to multinationality measures of international scientific collaboration," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(1), pages 75-89, April.
    12. Gualberto Buela-Casal & Pandelis Perakakis & Michael Taylor & Purificación Checa, 2006. "Measuring internationality: Reflections and perspectives on academic journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(1), pages 45-65, April.
    13. Ahlgren, Per & Waltman, Ludo, 2014. "The correlation between citation-based and expert-based assessments of publication channels: SNIP and SJR vs. Norwegian quality assessments," Journal of Informetrics, Elsevier, vol. 8(4), pages 985-996.
    14. Tianwei He & Wei Liu, 2009. "The internationalization of Chinese scientific journals: A quantitative comparison of three chemical journals from China, England and Japan," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(3), pages 583-593, September.
    15. Alan Sangster, 2015. "You Cannot Judge a Book by Its Cover: The Problems with Journal Rankings," Accounting Education, Taylor & Francis Journals, vol. 24(3), pages 175-186, June.
    16. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    17. Antonio Ferrara & Andrea Bonaccorsi, 2016. "How robust is journal rating in Humanities and Social Sciences? Evidence from a large-scale, multi-method exercise," Research Evaluation, Oxford University Press, vol. 25(3), pages 279-291.
    18. Saarela, Mirka & Kärkkäinen, Tommi & Lahtonen, Tommi & Rossi, Tuomo, 2016. "Expert-based versus citation-based ranking of scholarly and scientific publication channels," Journal of Informetrics, Elsevier, vol. 10(3), pages 693-718.
    19. Chiang Kao, 2009. "The authorship and internationality of Industrial Engineering journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(1), pages 123-136, October.
    20. Rey-Rocha Jesús & Martín-Sempere María José, 2004. "Patterns of the foreign contributions in some domestic vs. international journals on Earth Sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(1), pages 95-115, January.
    21. Gaby Haddow & Paul Genoni, 2010. "Citation analysis and peer ranking of Australian social science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 471-487, November.
    22. Ali Uzun, 2004. "Assessing internationality of scholarly journals through foreign authorship patterns: the case of major journals in information science, and scientometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 61(3), pages 457-465, November.
    23. Pajić, Dejan, 2015. "On the stability of citation-based journal rankings," Journal of Informetrics, Elsevier, vol. 9(4), pages 990-1006.
    24. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    25. Emanuela Reale & Anna Barbara & Antonio Costantini, 2007. "Peer review for the evaluation of academic research: lessons from the Italian experience," Research Evaluation, Oxford University Press, vol. 16(3), pages 216-228, September.
    26. Haddawy, Peter & Hassan, Saeed-Ul & Asghar, Awais & Amin, Sarah, 2016. "A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality," Journal of Informetrics, Elsevier, vol. 10(1), pages 162-173.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fabio Gomes Rocha & Rosimeri Ferraz Sabino & Alejandro C. Frery, 2020. "Analysis of the international impact of the Brazilian base “Qualis”-Education," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 1949-1963, December.
    2. Antonia Ferrer-Sapena & Susana Díaz-Novillo & Enrique A. Sánchez-Pérez, 2017. "Measuring Time-Dynamics and Time-Stability of Journal Rankings in Mathematics and Physics by Means of Fractional p -Variations," Publications, MDPI, vol. 5(3), pages 1-14, September.
    3. Tomislav Korman & Tomislav Malvić & Vasyl Lozynskyi & Zlatko Briševac & Zoran Kovač & Lidia Hrnčević & Krešimir Pavlić, 2024. "Mining and Mineral Processing Journals in the WoS and Their Rankings When Merging SCIEx and ESCI Databases—Case Study Based on the JCR 2022 Data," Publications, MDPI, vol. 12(1), pages 1-13, January.
    4. Henk F. Moed & Felix Moya-Anegon & Vicente Guerrero-Bote & Carmen Lopez-Illescas & Myroslava Hladchenko, 2021. "Bibliometric assessment of national scientific journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3641-3666, April.
    5. Przemysław Korytkowski & Emanuel Kulczycki, 2019. "Examining how country-level science policy shapes publication patterns: the case of Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1519-1543, June.
    6. Korytkowski, Przemyslaw & Kulczycki, Emanuel, 2019. "Publication counting methods for a national research evaluation exercise," Journal of Informetrics, Elsevier, vol. 13(3), pages 804-816.
    7. Emanuel Kulczycki & Ying Huang & Alesia A. Zuccala & Tim C. E. Engels & Antonio Ferrara & Raf Guns & Janne Pölönen & Gunnar Sivertsen & Zehra Taşkın & Lin Zhang, 2022. "Uses of the Journal Impact Factor in national journal rankings in China and Europe," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(12), pages 1741-1754, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.
    2. Serenko, Alexander & Dohan, Michael, 2011. "Comparing the expert survey and citation impact journal ranking methods: Example from the field of Artificial Intelligence," Journal of Informetrics, Elsevier, vol. 5(4), pages 629-648.
    3. Saarela, Mirka & Kärkkäinen, Tommi, 2020. "Can we automate expert-based journal rankings? Analysis of the Finnish publication indicator," Journal of Informetrics, Elsevier, vol. 14(2).
    4. Mike Thelwall & Kayvan Kousha & Meiko Makita & Mahshid Abdoli & Emma Stuart & Paul Wilson & Jonathan Levitt, 2023. "In which fields do higher impact journals publish higher quality articles?," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(7), pages 3915-3933, July.
    5. Chiang Kao, 2009. "The authorship and internationality of Industrial Engineering journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(1), pages 123-136, October.
    6. Kulczycki, Emanuel & Korzeń, Marcin & Korytkowski, Przemysław, 2017. "Toward an excellence-based research funding system: Evidence from Poland," Journal of Informetrics, Elsevier, vol. 11(1), pages 282-298.
    7. Johan Lyhagen & Per Ahlgren, 2020. "Uncertainty and the ranking of economics journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2545-2560, December.
    8. Saarela, Mirka & Kärkkäinen, Tommi & Lahtonen, Tommi & Rossi, Tuomo, 2016. "Expert-based versus citation-based ranking of scholarly and scientific publication channels," Journal of Informetrics, Elsevier, vol. 10(3), pages 693-718.
    9. Gualberto Buela-Casal & Olga Gutiérrez-Martínez & María Paz Bermúdez-Sánchez & Oscar Vadillo-Muñoz, 2007. "Comparative study of international academic rankings of universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 71(3), pages 349-365, June.
    10. Currie, Russell R. & Pandher, Gurupdesh S., 2020. "Finance journal rankings: Active scholar assessment revisited," Journal of Banking & Finance, Elsevier, vol. 111(C).
    11. Haddawy, Peter & Hassan, Saeed-Ul & Asghar, Awais & Amin, Sarah, 2016. "A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality," Journal of Informetrics, Elsevier, vol. 10(1), pages 162-173.
    12. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    13. Jerome K. Vanclay & Lutz Bornmann, 2012. "Metrics to evaluate research performance in academic institutions: a critique of ERA 2010 as applied in forestry and the indirect H2 index as a possible alternative," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 751-771, June.
    14. Sumiko Asai, 2020. "The effect of collaboration with large publishers on the internationality and influence of open access journals for research institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 663-677, July.
    15. Sumiko Asai, 2021. "Collaboration between research institutes and large and small publishers for publishing open access journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 5245-5262, June.
    16. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    17. Hyeonchae Yang & Woo-Sung Jung, 2015. "A strategic management approach for Korean public research institutes based on bibliometric investigation," Quality & Quantity: International Journal of Methodology, Springer, vol. 49(4), pages 1437-1464, July.
    18. Bonaccorsi, Andrea & Haddawy, Peter & Cicero, Tindaro & Hassan, Saeed-Ul, 2017. "The solitude of stars. An analysis of the distributed excellence model of European universities," Journal of Informetrics, Elsevier, vol. 11(2), pages 435-454.
    19. Alberto Baccini & Lucio Barabesi & Giuseppe De Nicolao, 2020. "On the agreement between bibliometrics and peer review: Evidence from the Italian research assessment exercises," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-28, November.
    20. Rebora, Gianfranco & Turri, Matteo, 2013. "The UK and Italian research assessment exercises face to face," Research Policy, Elsevier, vol. 42(9), pages 1657-1666.

    More about this item

    Keywords

    Journal ranking; Multidimensional evaluation; Structural equation modelling; Expert-based evaluation; Citations; Poland;
    All these keywords.

    JEL classification:

    • O38 - Economic Development, Innovation, Technological Change, and Growth - - Innovation; Research and Development; Technological Change; Intellectual Property Rights - - - Government Policy

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:111:y:2017:i:1:d:10.1007_s11192-017-2261-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.