IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v86y2011i3d10.1007_s11192-010-0312-7.html
   My bibliography  Save this article

Overturning some assumptions about the effects of evaluation systems on publication performance

Author

Listed:
  • Carmen Osuna

    (CSIC Institute of Public Goods and Policies (IPP-CCHS), Consejo Superior de Investigaciones Científicas (CSIC))

  • Laura Cruz-Castro

    (CSIC Institute of Public Goods and Policies (IPP-CCHS), Consejo Superior de Investigaciones Científicas (CSIC))

  • Luis Sanz-Menéndez

    (CSIC Institute of Public Goods and Policies (IPP-CCHS), Consejo Superior de Investigaciones Científicas (CSIC))

Abstract

In 1989 the Spanish Government established an individual retrospective research evaluation system (RES) for public researchers. Policy makers have associated the establishment of this evaluation system with the significant increase in the volume of scientific publications attributed to Spain over the last decades. In a similar vein to the analyses of other country cases, some scholars have also claimed that the growth of Spain’s international scientific publications is a result of the establishment of the new evaluation system. In this paper, we provide a methodological revision of the validity threats in previous research, including some interrupted time-series analyses and control groups to investigate the effects of this policy instrument on the number of papers produced by Spanish authors. In the years following the establishment of the evaluation system, the results indicate a considerable increase in the number of papers attributed to Spanish authors among those eligible for evaluation (the “treated” group), but also in the control groups. After testing various alternative explanations, we conclude that the growth in Spanish publications cannot be attributed indisputably to the effect of the establishment of the RES, but rather to the increase of expenditure and number of researchers in the Spanish R&D system along with some maturation effects. We take this case as an example of the need to improve and refine methodologies and to be more cautious when attributing effects to research evaluation mechanisms at the national level.

Suggested Citation

  • Carmen Osuna & Laura Cruz-Castro & Luis Sanz-Menéndez, 2011. "Overturning some assumptions about the effects of evaluation systems on publication performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(3), pages 575-592, March.
    • Handle: RePEc:spr:scient:v:86:y:2011:i:3:d:10.1007_s11192-010-0312-7
    DOI: 10.1007/s11192-010-0312-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-010-0312-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-010-0312-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Robert D. Shelton, 2008. "Relations between national research investment and publication output: Application to an American Paradox," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 191-205, February.
    2. Çetin Önder & Mehmet Sevkli & Taner Altinok & Cengiz Tavukçuoǧlu, 2008. "Institutional change and scientific research: A preliminary bibliometric analysis of institutional influences on Turkey’s recent social science publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(3), pages 543-560, September.
    3. Henk F. Moed, 2008. "UK Research Assessment Exercises: Informed judgments on research quality or quantity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(1), pages 153-161, January.
    4. F. Moya-Anegón & V. Herrero-Solana, 1999. "Science in america latina: A comparison of bibliometric and scientific-technical indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(2), pages 299-320, October.
    5. H F Moed & Th N van Leeuwen & M S Visser, 1999. "Trends in publication output and impact of universities in the Netherlands," Research Evaluation, Oxford University Press, vol. 8(1), pages 60-67, April.
    6. Linda Butler, 2003. "Modifying publication practices in response to funding formulas," Research Evaluation, Oxford University Press, vol. 12(1), pages 39-46, April.
    7. Whitley, Richard, 2003. "Competition and pluralism in the public sciences: the impact of institutional frameworks on the organisation of academic science," Research Policy, Elsevier, vol. 32(6), pages 1015-1029, June.
    8. Crespi, Gustavo A. & Geuna, Aldo, 2008. "An empirical study of scientific production: A cross country analysis, 1981-2002," Research Policy, Elsevier, vol. 37(4), pages 565-579, May.
    9. Olle Persson & Wolfgang Glänzel & Rickard Danell, 2004. "Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(3), pages 421-432, August.
    10. Peter Ingwersen & Daisy Jacobs, 2004. "South African research in selected scientific areas: Status 1981–2000," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(3), pages 405-423, March.
    11. Mohr, L. B., 2000. "Regression artifacts and other customs of dubious desert," Evaluation and Program Planning, Elsevier, vol. 23(4), pages 397-409, November.
    12. Gomez, I. & Fernandez, M. T. & Zulueta, M. A. & Cami, J., 1995. "Analysis of biomedical research in Spain," Research Policy, Elsevier, vol. 24(3), pages 459-471, May.
    13. Anthony F J van Raan, 1997. "Science as an international enterprise," Science and Public Policy, Oxford University Press, vol. 24(5), pages 290-300, October.
    14. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yaşar Tonta & Müge Akbulut, 2020. "Does monetary support increase citation impact of scholarly papers?," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1617-1641, November.
    2. Irene López-Navarro & Ana I. Moreno & Miguel Ángel Quintanilla & Jesús Rey-Rocha, 2015. "Why do I publish research articles in English instead of my own language? Differences in Spanish researchers’ motivations across scientific domains," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 939-976, June.
    3. M. Teresa Antonio-García & Irene López-Navarro & Jesús Rey-Rocha, 2014. "Determinants of success for biomedical researchers: a perception-based study in a health science research environment," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(3), pages 1747-1779, December.
    4. Gabriel-Alexandru Vîiu & Mihai Păunescu, 2021. "The citation impact of articles from which authors gained monetary rewards based on journal metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4941-4974, June.
    5. Daniele Fanelli, 2012. "Negative results are disappearing from most disciplines and countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 891-904, March.
    6. Luis Sanz-Menéndez & Laura Cruz-Castro & Kenedy Alva, 2013. "Time to Tenure in Spanish Universities: An Event History Analysis," PLOS ONE, Public Library of Science, vol. 8(10), pages 1-1, October.
    7. Ramón A. Feenstra & Emilio Delgado López-Cózar, 2022. "Philosophers’ appraisals of bibliometric indicators and their use in evaluation: from recognition to knee-jerk rejection," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 2085-2103, April.
    8. Linda Sīle & Raf Vanderstraeten, 2019. "Measuring changes in publication patterns in a context of performance-based research funding systems: the case of educational research in the University of Gothenburg (2005–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 71-91, January.
    9. Selcuk Besir Demir, 2018. "Pros and cons of the new financial support policy for Turkish researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2053-2068, September.
    10. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    11. Antonio Fernández-Cano & Manuel Torralbo & Mónica Vallejo, 2012. "Time series of scientific growth in Spanish doctoral theses (1848–2009)," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(1), pages 15-36, April.
    12. Alonso Rodríguez-Navarro & Ricardo Brito, 2022. "The link between countries’ economic and scientific wealth has a complex dependence on technological activity and research policy," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2871-2896, May.
    13. van den Besselaar, Peter & Heyman, Ulf & Sandström, Ulf, 2017. "Perverse effects of output-based research funding? Butler’s Australian case revisited," Journal of Informetrics, Elsevier, vol. 11(3), pages 905-918.
    14. Artés, Joaquín & Pedraja-Chaparro, Francisco & Salinas-JiméneZ, Mª del Mar, 2017. "Research performance and teaching quality in the Spanish higher education system: Evidence from a medium-sized university," Research Policy, Elsevier, vol. 46(1), pages 19-29.
    15. Pin-Hua Lin & Jong-Rong Chen & Chih-Hai Yang, 2014. "Academic research resources and academic quality: a cross-country analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 109-123, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Carmen Osuna & Laura Cruz Castro & Luis Sanz Menéndez, 2010. "Knocking down some Assumptions about the Effects of Evaluation Systems on Publications," Working Papers 1010, Instituto de Políticas y Bienes Públicos (IPP), CSIC.
    2. Alonso Rodríguez-Navarro & Francis Narin, 2018. "European Paradox or Delusion—Are European Science and Economy Outdated?," Science and Public Policy, Oxford University Press, vol. 45(1), pages 14-23.
    3. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    4. Marianne Gauffriau & Peder Olesen Larsen & Isabelle Maye & Anne Roulin-Perriard & Markus Ins, 2007. "Publication, cooperation and productivity measures in scientific research," Scientometrics, Springer;Akadémiai Kiadó, vol. 73(2), pages 175-214, November.
    5. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    6. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    7. Auranen, Otto & Nieminen, Mika, 2010. "University research funding and publication performance--An international comparison," Research Policy, Elsevier, vol. 39(6), pages 822-834, July.
    8. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "A national-scale cross-time analysis of university research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(2), pages 399-413, May.
    9. van den Besselaar, Peter & Heyman, Ulf & Sandström, Ulf, 2017. "Perverse effects of output-based research funding? Butler’s Australian case revisited," Journal of Informetrics, Elsevier, vol. 11(3), pages 905-918.
    10. Linda Sīle & Raf Vanderstraeten, 2019. "Measuring changes in publication patterns in a context of performance-based research funding systems: the case of educational research in the University of Gothenburg (2005–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 71-91, January.
    11. Yuret, Tolga, 2017. "Do researchers pay attention to publication subsidies?," Journal of Informetrics, Elsevier, vol. 11(2), pages 423-434.
    12. Abramo, Giovanni & D'Angelo, CiriacoAndrea & Di Costa, Flavia, 2024. "The moderating role of personal characteristics of authors in the publications’ quality for quantity trade-off," Journal of Informetrics, Elsevier, vol. 18(1).
    13. Sandström, Ulf & Van den Besselaar, Peter, 2018. "Funding, evaluation, and the performance of national research systems," Journal of Informetrics, Elsevier, vol. 12(1), pages 365-384.
    14. R. Inglesi-Lotz & A. Pouris, 2013. "The influence of scientific research output of academics on economic growth in South Africa: an autoregressive distributed lag (ARDL) application," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 129-139, April.
    15. Lutz Bornmann, 2011. "Mimicry in science?," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(1), pages 173-177, January.
    16. Max Leckert, 2021. "(E-) Valuative Metrics as a Contested Field: A Comparative Analysis of the Altmetrics- and the Leiden Manifesto," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9869-9903, December.
    17. Elham Erfanian & Amir B. Ferreira Neto, 2017. "Scientific output: labor or capital intensive? An analysis for selected countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(1), pages 461-482, July.
    18. Selcuk Besir Demir, 2018. "Pros and cons of the new financial support policy for Turkish researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2053-2068, September.
    19. Marianne Gauffriau & Peder Olesen Larsen & Isabelle Maye & Anne Roulin-Perriard & Markus Ins, 2008. "Comparisons of results of publication counting using different methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 77(1), pages 147-176, October.
    20. Pin-Hua Lin & Jong-Rong Chen & Chih-Hai Yang, 2014. "Academic research resources and academic quality: a cross-country analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 109-123, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:86:y:2011:i:3:d:10.1007_s11192-010-0312-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.