IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v125y2020i1d10.1007_s11192-020-03617-z.html
   My bibliography  Save this article

Evaluation of Brazilian research output in education: confronting international and national contexts

Author

Listed:
  • Eliseo Reategui

    (Universidade Federal do Rio Grande do Sul)

  • Alause Pires

    (Universidade Federal do Rio Grande do Sul)

  • Michel Carniato
  • Sergio Roberto Kieling Franco

    (Universidade Federal do Rio Grande do Sul)

Abstract

Scientific production has increased considerably in the last two decades in Brazil, placing the country as an emerging scientific power. In the field of Education, the same can be observed, with a significant growth in the number of articles published by Brazilian researchers in the recent past. In this article we evaluate Brazilian research output in Education from 2007 to 2016, with the intent to understand how national evaluation standards compare with international parameters dictated by bibliometric indicators. We confront the citation impact of Brazilian publications in the period with their classification in an expert-based journal evaluation system called QUALIS. We used Scopus’ SNIP bibliometric indicator for this analysis. The study was carried out using data about 40,825 articles published in 2719 different journals. Results showed that only a small percentage of these articles featured in Scopus indexed journals (13.28%), and most of these journals were published in Brazil (66%). The citation impact of the Scopus indexed publications had a significant growth in the period, but journals with dissimilar citation impact were not distinctively distributed in separate QUALIS categories. These findings show a certain publishing pattern that is likely to be related to the association of the Brazilian research evaluation and funding systems. In addition, they raise questions about how the establishment of evaluation criteria that is mainly subjective and does not include specific metrics may hinder the visibility of research output from a global perspective.

Suggested Citation

  • Eliseo Reategui & Alause Pires & Michel Carniato & Sergio Roberto Kieling Franco, 2020. "Evaluation of Brazilian research output in education: confronting international and national contexts," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 427-444, October.
    • Handle: RePEc:spr:scient:v:125:y:2020:i:1:d:10.1007_s11192-020-03617-z
    DOI: 10.1007/s11192-020-03617-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03617-z
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-020-03617-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2011. "Evaluating research: from informed peer review to bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 499-514, June.
    2. Richard Van Noorden, 2014. "The impact gap: South America by the numbers," Nature, Nature, vol. 510(7504), pages 202-203, June.
    3. Vincent Larivière & Benoit Macaluso, 2011. "Improving the coverage of social science and humanities researchers' output: The case of the Érudit journal platform," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(12), pages 2437-2442, December.
    4. Stephen A Gallo & Joanne H Sullivan & Scott R Glisson, 2016. "The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications," PLOS ONE, Public Library of Science, vol. 11(10), pages 1-18, October.
    5. Joshua S. Gans & George B. Shepherd, 1994. "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 165-179, Winter.
    6. Chavarro, Diego & Tang, Puay & Ràfols, Ismael, 2017. "Why researchers publish in non-mainstream journals: Training, knowledge bridging, and gap filling," Research Policy, Elsevier, vol. 46(9), pages 1666-1680.
    7. Francisco Collazo-Reyes, 2014. "Growth of the number of indexed journals of Latin America and the Caribbean: the effect on the impact of each country," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 197-209, January.
    8. Yun-Wen Chuang & Ling-Chu Lee & Wen-Chi Hung & Pin-Hua Lin, 2010. "Forging Into The Innovation Lead — A Comparative Analysis Of Scientific Capacity," International Journal of Innovation Management (ijim), World Scientific Publishing Co. Pte. Ltd., vol. 14(03), pages 511-529.
    9. Maria Bordons & M. T. Fernández & Isabel Gómez, 2002. "Advantages and limitations in the use of impact factor measures for the assessment of research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 195-206, February.
    10. Emanuela Reale & Anna Barbara & Antonio Costantini, 2007. "Peer review for the evaluation of academic research: lessons from the Italian experience," Research Evaluation, Oxford University Press, vol. 16(3), pages 216-228, September.
    11. Daniel Vaughan-Whitehead, 2016. "Introduction," Economia & lavoro, Carocci editore, issue 2, pages 7-12.
    12. Gunnar Sivertsen, 2016. "Patterns of internationalization and criteria for research assessment in the social sciences and humanities," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 357-368, May.
    13. Kendall Powell, 2012. "Publishing: Foreign tongues," Nature, Nature, vol. 487(7405), pages 129-131, July.
    14. André Frazão Helene & Pedro Leite Ribeiro, 2011. "Brazilian scientific production, financial support, established investigators and doctoral graduates," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(2), pages 677-686, November.
    15. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S., 2013. "Some modifications to the SNIP journal impact indicator," Journal of Informetrics, Elsevier, vol. 7(2), pages 272-285.
    16. Éric Archambault & Vincent Larivière, 2009. "History of the journal impact factor: Contingencies and consequences," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(3), pages 635-649, June.
    17. Vincent Larivière & Benoit Macaluso, 2011. "Improving the coverage of social science and humanities researchers' output: The case of the Érudit journal platform," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(12), pages 2437-2442, December.
    18. Diana Hicks & Jian Wang, 2011. "Coverage and overlap of the new social sciences and humanities journal lists," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(2), pages 284-294, February.
    19. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    20. Truyken L. B. Ossenblok & Tim C. E. Engels & Gunnar Sivertsen, 2012. "The representation of the social sciences and humanities in the Web of Science--a comparison of publication patterns and incentive structures in Flanders and Norway (2005--9)," Research Evaluation, Oxford University Press, vol. 21(4), pages 280-290, September.
    21. Diana Hicks & Jian Wang, 2011. "Coverage and overlap of the new social sciences and humanities journal lists," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 284-294, February.
    22. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    23. Day, Theodore Eugene, 2015. "The big consequences of small biases: A simulation of peer review," Research Policy, Elsevier, vol. 44(6), pages 1266-1270.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Elena Veretennik & Maria Yudkevich, 2023. "Inconsistent quality signals: evidence from the regional journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3675-3701, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    3. Gordana Budimir & Sophia Rahimeh & Sameh Tamimi & Primož Južnič, 2021. "Comparison of self-citation patterns in WoS and Scopus databases based on national scientific production in Slovenia (1996–2020)," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 2249-2267, March.
    4. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    5. McAleer, M.J. & Oláh, J. & Popp, J., 2018. "Pros and Cons of the Impact Factor in a Rapidly Changing Digital World," Econometric Institute Research Papers EI2018-11, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.
    6. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    7. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    8. Chavarro, Diego & Tang, Puay & Ràfols, Ismael, 2017. "Why researchers publish in non-mainstream journals: Training, knowledge bridging, and gap filling," Research Policy, Elsevier, vol. 46(9), pages 1666-1680.
    9. Pei-Shan Chi, 2014. "Which role do non-source items play in the social sciences? A case study in political science in Germany," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1195-1213, November.
    10. Daniela Filippo & Rafael Aleixandre-Benavent & Elías Sanz-Casado, 2020. "Toward a classification of Spanish scholarly journals in social sciences and humanities considering their impact and visibility," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1709-1732, November.
    11. Bryce, Cormac & Dowling, Michael & Lucey, Brian, 2020. "The journal quality perception gap," Research Policy, Elsevier, vol. 49(5).
    12. Marco Cozzi, 2020. "Public Funding of Research and Grant Proposals in the Social Sciences: Empirical Evidence from Canada," Department Discussion Papers 1809, Department of Economics, University of Victoria.
    13. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2023. "Correlating article citedness and journal impact: an empirical investigation by field on a large-scale dataset," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1877-1894, March.
    14. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    15. Lehmann, Robert & Wohlrabe, Klaus, 2017. "Who is the ‘Journal Grand Master’? A new ranking based on the Elo rating system," Journal of Informetrics, Elsevier, vol. 11(3), pages 800-809.
    16. Antonin Mac'e, 2017. "The Limits of Citation Counts," Papers 1711.02695, arXiv.org, revised Sep 2023.
    17. Tingcan Ma & Gui-Fang Wang & Ke Dong & Mukun Cao, 2012. "The Journal’s Integrated Impact Index: a new indicator for journal evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(2), pages 649-658, February.
    18. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    19. B Ian Hutchins & Xin Yuan & James M Anderson & George M Santangelo, 2016. "Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level," PLOS Biology, Public Library of Science, vol. 14(9), pages 1-25, September.
    20. Xinning Su & Sanhong Deng & Si Shen, 2014. "The design and application value of the Chinese Social Science Citation Index," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 1567-1582, March.

    More about this item

    Keywords

    Research assessment; Education; QUALIS; CAPES; Brazil;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:125:y:2020:i:1:d:10.1007_s11192-020-03617-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.