IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v128y2023i6d10.1007_s11192-023-04723-4.html
   My bibliography  Save this article

Inconsistent quality signals: evidence from the regional journals

Author

Listed:
  • Elena Veretennik

    (HSE University)

  • Maria Yudkevich

    (HSE University)

Abstract

Nowadays many countries and institutions use bibliometric assessment of journal quality in their research evaluation policies. However, bibliometric measures, such as impact factor or quartile, may provide a biased quality assessment for relatively new, regional, or non-mainstream journals, as these outlets usually do not possess a longstanding history, and may not be included into indexing databases. To reduce the information asymmetry between academic community (researchers, editors, policymakers) and journal management, we propose an alternative approach to evaluate journals quality signals using previous publication track record of authors. We explore the difference in the quality signals sent by regional journals. Traditional, journal-level, bibliometric measures are contrasted with generalised measures of authors' publishing records. We used a set of 50,477 articles and reviews in 83 regional journals in Physics and Astronomy (2014–2019) to extract and process data on 73 866 authors and their additional 329,245 publications in other Scopus-indexed journals. We found that traditional journal-level measures (such as journal quartile, CiteScore percentile, Scimago Journal Rank) tend to under-evaluate journal quality, thus contributing to an image of low-quality research venues. Author-level measures (including the share of papers in the Nature Index journals) send positive signals of journal quality and allow us to subdivide regional journals by their publishing strategies. These results suggest that research evaluation policies might consider attributing greater weight to regional journals, not only for the training purposes of doctoral students but also for gaining international visibility and impact.

Suggested Citation

  • Elena Veretennik & Maria Yudkevich, 2023. "Inconsistent quality signals: evidence from the regional journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3675-3701, June.
    • Handle: RePEc:spr:scient:v:128:y:2023:i:6:d:10.1007_s11192-023-04723-4
    DOI: 10.1007/s11192-023-04723-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-023-04723-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-023-04723-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Sascha Baghestanian & Sergey V. Popov, 2018. "On publication, refereeing and working hard," Canadian Journal of Economics, Canadian Economics Association, vol. 51(4), pages 1419-1459, November.
    2. Diego Chavarro & Ismael Ràfols & Puay Tang, 2018. "To what extent is inclusion in the Web of Science an indicator of journal ‘quality’?," Research Evaluation, Oxford University Press, vol. 27(3), pages 284-284.
    3. Aurelia Magdalena Pisoschi & Claudia Gabriela Pisoschi, 2016. "Is open access the solution to increase the impact of scientific journals?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1075-1095, November.
    4. Riley, John G., 1975. "Competitive signalling," Journal of Economic Theory, Elsevier, vol. 10(2), pages 174-186, April.
    5. Tudorel Andrei & Daniel Teodorescu & Andreea Mirică, 2016. "Beyond the Impact Factor: measuring the international visibility of Romanian social sciences journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 1-20, July.
    6. Moed, Henk F. & de Moya-Anegon, Felix & Guerrero-Bote, Vicente & Lopez-Illescas, Carmen, 2020. "Are nationally oriented journals indexed in Scopus becoming more international? The effect of publication language and access modality," Journal of Informetrics, Elsevier, vol. 14(2).
    7. Henk F. Moed & Felix Moya-Anegon & Vicente Guerrero-Bote & Carmen Lopez-Illescas & Myroslava Hladchenko, 2021. "Bibliometric assessment of national scientific journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3641-3666, April.
    8. Michaël Bikard, 2018. "Made in Academia: The Effect of Institutional Origin on Inventors’ Attention to Science," Organization Science, INFORMS, vol. 29(5), pages 818-836, October.
    9. Engers, Maxim, 1987. "Signalling with Many Signals," Econometrica, Econometric Society, vol. 55(3), pages 663-674, May.
    10. Michaël Bikard & Matt Marx, 2020. "Bridging Academia and Industry: How Geographic Hubs Connect University Science and Corporate Technology," Management Science, INFORMS, vol. 66(8), pages 3425-3443, August.
    11. Emanuel Kulczycki & Ying Huang & Alesia A. Zuccala & Tim C. E. Engels & Antonio Ferrara & Raf Guns & Janne Pölönen & Gunnar Sivertsen & Zehra Taşkın & Lin Zhang, 2022. "Uses of the Journal Impact Factor in national journal rankings in China and Europe," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(12), pages 1741-1754, December.
    12. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2021. "The interplay between the reviewer’s incentives and the journal’s quality standard," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3041-3061, April.
    13. Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    14. Sugimoto, Cassidy R. & Larivière, Vincent & Ni, Chaoqun & Cronin, Blaise, 2013. "Journal acceptance rates: A cross-disciplinary analysis of variability and relationships with journal measures," Journal of Informetrics, Elsevier, vol. 7(4), pages 897-906.
    15. Ying Huang & Donghua Zhu & Qi Lv & Alan L. Porter & Douglas K. R. Robinson & Xuefeng Wang, 2017. "Early insights on the Emerging Sources Citation Index (ESCI): an overlay map-based bibliometric study," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 2041-2057, June.
    16. David Michayluk & Ralf Zurbruegg, 2014. "Do lead articles signal higher quality in the digital age? Evidence from finance journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 961-973, February.
    17. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    18. Marshall Medoff, 2003. "Article placement and market signalling," Applied Economics Letters, Taylor & Francis Journals, vol. 10(8), pages 479-482.
    19. Maia Chankseliani & Andrey Lovakov & Vladimir Pislyakov, 2021. "A big picture: bibliometric study of academic publications from post-Soviet countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(10), pages 8701-8730, October.
    20. Chris Fields, 2015. "Co-authorship proximity of A. M. Turing Award and John von Neumann Medal winners to the disciplinary boundaries of computer science," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 809-825, September.
    21. Eliseo Reategui & Alause Pires & Michel Carniato & Sergio Roberto Kieling Franco, 2020. "Evaluation of Brazilian research output in education: confronting international and national contexts," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 427-444, October.
    22. Myroslava Hladchenko & Henk F. Moed, 2021. "National orientation of Ukrainian journals: means-ends decoupling in a semi-peripheral state," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 2365-2389, March.
    23. José Alonso Borba & Alessanderson Jacó Carvalho & Denize Demarche Minatti Ferreira & Fábio Minatto, 2021. "Scopes of accounting journals and published papers: what do they signalize?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5665-5685, July.
    24. Ryazanova, Olga & McNamara, Peter & Aguinis, Herman, 2017. "Research performance as a quality signal in international labor markets: Visibility of business schools worldwide through a global research performance system," Journal of World Business, Elsevier, vol. 52(6), pages 831-841.
    25. Andrey Lovakov & Anna Panova & Maria Yudkevich, 2022. "Global visibility of nationally published research output: the case of the post-Soviet region," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2643-2659, May.
    26. Hendrik P. van Dalen & K?ne Henkens, 2005. "Signals in science - On the importance of signaling in gaining attention in science," Scientometrics, Springer;Akadémiai Kiadó, vol. 64(2), pages 209-233, August.
    27. Didegah, Fereshteh & Thelwall, Mike, 2013. "Which factors help authors produce the highest impact research? Collaboration, journal and document properties," Journal of Informetrics, Elsevier, vol. 7(4), pages 861-873.
    28. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    29. Leonid Tiokhin & Karthik Panchanathan & Daniel Lakens & Simine Vazire & Thomas Morgan & Kevin Zollman, 2021. "Honest signaling in academic publishing," PLOS ONE, Public Library of Science, vol. 16(2), pages 1-19, February.
    30. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    31. Marcela Suárez-Tamayo & Francisco Collazo-Reyes & Miguel Ángel Pérez-Angón, 2018. "Emerging roles of regional journals in the accreditation of knowledge in tropical medicine: Biomédica and Memorias do Instituto Oswaldo Cruz, 2007–2015," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1615-1625, June.
    32. Dag W Aksnes, 2003. "Characteristics of highly cited papers," Research Evaluation, Oxford University Press, vol. 12(3), pages 159-170, December.
    33. M. H. Medoff, 2003. "Article placement and market signalling," Applied Economics Letters, Taylor & Francis Journals, vol. 10(10), pages 601-604.
    34. Joshua S. Gans & George B. Shepherd, 1994. "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 165-179, Winter.
    35. Michael Spence, 1973. "Job Market Signaling," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 87(3), pages 355-374.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rüdiger Mutz & Tobias Wolbring & Hans-Dieter Daniel, 2017. "The effect of the “very important paper” (VIP) designation in Angewandte Chemie International Edition on citation impact: A propensity score matching analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2139-2153, September.
    2. Andreea Mironescu & Alina Moroșanu & Anca-Diana Bibiri, 2023. "The regional dynamics of multilingual publishing in web of science: A statistical analysis of central and eastern european journals and researchers in linguistics," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(2), pages 1133-1162, February.
    3. Meyer, Matthias & Waldkirch, Rüdiger W. & Duscher, Irina & Just, Alexander, 2018. "Drivers of citations: An analysis of publications in “top” accounting journals," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 51(C), pages 24-46.
    4. Nasirov, Shukhrat & Joshi, Amol M., 2023. "Minding the communications gap: How can universities signal the availability and value of their scientific knowledge to commercial organizations?," Research Policy, Elsevier, vol. 52(9).
    5. Besancenot, Damien & Vranceanu, Radu, 2024. "Reluctance to pursue breakthrough research: A signaling explanation," Research Policy, Elsevier, vol. 53(4).
    6. David Michayluk & Ralf Zurbruegg, 2014. "Do lead articles signal higher quality in the digital age? Evidence from finance journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 961-973, February.
    7. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    8. Jens Jirschitzka & Aileen Oeberst & Richard Göllner & Ulrike Cress, 2017. "Inter-rater reliability and validity of peer reviews in an interdisciplinary field," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 1059-1092, November.
    9. Petersen, Alexander M., 2019. "Megajournal mismanagement: Manuscript decision bias and anomalous editor activity at PLOS ONE," Journal of Informetrics, Elsevier, vol. 13(4).
    10. Vicente Safón & Domingo Docampo, 2020. "Analyzing the impact of reputational bias on global university rankings based on objective research performance data: the case of the Shanghai Ranking (ARWU)," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2199-2227, December.
    11. Bayar, Onur & Chemmanur, Thomas J., 2021. "A model of the editorial process in academic journals," Research Policy, Elsevier, vol. 50(9).
    12. Andrey Lovakov & Anna Panova & Maria Yudkevich, 2022. "Global visibility of nationally published research output: the case of the post-Soviet region," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2643-2659, May.
    13. Vicente Safón, 2019. "Inter-ranking reputational effects: an analysis of the Academic Ranking of World Universities (ARWU) and the Times Higher Education World University Rankings (THE) reputational relationship," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 897-915, November.
    14. Libo Sheng & Dongqing Lyu & Xuanmin Ruan & Hongquan Shen & Ying Cheng, 2023. "The association between prior knowledge and the disruption of an article," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4731-4751, August.
    15. Sabatier, Mareva & Chollet, Barthélemy, 2017. "Is there a first mover advantage in science? Pioneering behavior and scientific production in nanotechnology," Research Policy, Elsevier, vol. 46(2), pages 522-533.
    16. Mingyang Wang & Shi Li & Guangsheng Chen, 2017. "Detecting latent referential articles based on their vitality performance in the latest 2 years," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1557-1571, September.
    17. Thelwall, Mike & Kousha, Kayvan & Stuart, Emma & Makita, Meiko & Abdoli, Mahshid & Wilson, Paul & Levitt, Jonathan, 2023. "Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?," Research Policy, Elsevier, vol. 52(8).
    18. Naiditch, Claire & Vranceanu, Radu, 2011. "Remittances as a social status signaling device," Research in Economics, Elsevier, vol. 65(4), pages 305-318, December.
    19. Jürgen Janger & Nicole Schmidt-Padickakudy & Anna Strauss-Kollin, 2019. "International Differences in Basic Research Grant Funding. A Systematic Comparison," WIFO Studies, WIFO, number 61664, March.
    20. Andrés Marroquín & Julio H. Cole, 2015. "Economical writing (or, “Think Hemingway”)," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(1), pages 251-259, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:128:y:2023:i:6:d:10.1007_s11192-023-04723-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.