IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v88y2011i3d10.1007_s11192-011-0440-8.html
   My bibliography  Save this article

Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa

Author

Listed:
  • Roula Inglesi-Lotz

    (University of Pretoria)

  • Anastassios Pouris

    (University of Pretoria)

Abstract

The influence of the National Research Foundation’s (NRF) rating system on the productivity of the South African social science researchers is investigated scientometrically for the period from 1981 to 2006. Their output performance is mainly indicated by their research publications. Following international best practice in scientometrics as well as the behavioural reinforcement theory, we employed the “before/after control impact (BACI) method”, as well as the well known econometric breakpoint test as proposed by Chow. We use as control group the publications in the field of clinical medicine. The field is not supported by NRF and hence clinical medicine researchers are not affected by the evaluation and rating system. The findings show a positive impact of the NRF programme on the research outputs of social sciences researchers and the implementation of the programme has increased the relevant population of research articles by an average of 24.5% (during the first 5 years) over the expected number of publication without the programme. The results confirm the scientometric findings of other studies (e.g. that of Nederhof) that ratings promulgate research productivity.

Suggested Citation

  • Roula Inglesi-Lotz & Anastassios Pouris, 2011. "Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 747-760, September.
    • Handle: RePEc:spr:scient:v:88:y:2011:i:3:d:10.1007_s11192-011-0440-8
    DOI: 10.1007/s11192-011-0440-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-011-0440-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-011-0440-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marianne Bertrand & Simeon Djankov & Rema Hanna & Sendhil Mullainathan, 2006. "Does corruption produce unsafe drivers?," Natural Field Experiments 00218, The Field Experiments Website.
    2. Sandra Miguel & Félix Moya-Anegón & Víctor Herrero-Solana, 2010. "The impact of the socio-economic crisis of 2001 on the scientific system of Argentina from the scientometric perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 495-507, November.
    3. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    4. Andrew Leigh, 2003. "Employment Effects of Minimum Wages: Evidence from a Quasi‐Experiment," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 36(4), pages 361-373, December.
    5. Anton J. Nederhof, 2008. "Policy impact of bibliometric rankings of research performance of departments and individuals in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(1), pages 163-174, January.
    6. Mee-Jean Kim, 2007. "A bibliometric analysis of the effectiveness of Korea’s Biotechnology Stimulation Plans, with a comparison with four other Asian nations," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(3), pages 371-388, September.
    7. David A Raitzer & Timothy G Kelley, 2008. "Assessing the contribution of impact assessment to donor decisions for international agricultural research," Research Evaluation, Oxford University Press, vol. 17(3), pages 187-199, September.
    8. Michael Kremer, 2003. "Randomized Evaluations of Educational Programs in Developing Countries: Some Lessons," American Economic Review, American Economic Association, vol. 93(2), pages 102-106, May.
    9. Groot, Tom & Garcia-Valderrama, Teresa, 2006. "Research quality and efficiency: An analysis of assessments and management issues in Dutch economics and business research programs," Research Policy, Elsevier, vol. 35(9), pages 1362-1376, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Roula Inglesi-Lotz & Mehmet Balcilar & Rangan Gupta, 2014. "Time-varying causality between research output and economic growth in US," Scientometrics, Springer;Akadémiai Kiadó, vol. 100(1), pages 203-216, July.
    2. Fernando A. B. Colugnati & Sergio Firpo & Paula F. Drummond Castro & Juan E. Sepulveda & Sergio L. M. Salles-Filho, 2014. "A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 85-107, October.
    3. Nelson Casimiro Zavale & Patrício Vitorino Langa, 2018. "University-industry linkages’ literature on Sub-Saharan Africa: systematic literature review and bibliometric account," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 1-49, July.
    4. Anastassios Pouris, 2012. "Scientometric research in South Africa and successful policy instruments," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 317-325, May.
    5. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    6. R. Inglesi-Lotz & A. Pouris, 2013. "The influence of scientific research output of academics on economic growth in South Africa: an autoregressive distributed lag (ARDL) application," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 129-139, April.
    7. Similo Ngwenya & Nelius Boshoff, 2022. "Different manifestations of ‘context’: examples from a bibliometric study of research in Zimbabwe in Southern Africa," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 3911-3933, July.
    8. Fiorentin, Florencia & Pereira, Mariano & Suárez, Diana, 2022. "The Gender Gap in Public S&T Funding: The Matilda Effect in STEM Disciplines in Argentina," IDB Publications (Working Papers) 11990, Inter-American Development Bank.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    2. Sawada Yasuyuki & Mahmud Minhaj & Seki Mai & Le An & Kawarazaki Hikaru, 2017. "Individualized Self-learning Program to Improve Primary Education: Evidence from a Randomized Field Experiment in Bangladesh," Working Papers 156, JICA Research Institute.
    3. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    4. Committee, Nobel Prize, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    5. Clarke, Damian, 2019. "A convenient omitted variable bias formula for treatment effect models," Economics Letters, Elsevier, vol. 174(C), pages 84-88.
    6. Sawada, Yasuyuki & Ishii, Takaharu, 2012. "Do Community-Managed Schools Facilitate Social Capital Accumulation? Evidence from the COGES Project in Burkina Faso," Working Papers 42, JICA Research Institute.
    7. Emmanuel Jimenez & Yasuyuki Sawada, 2014. "Does Community Management Help Keep Children in Schools? Evidence Using Panel Data from El Salvador's EDUCO Program," Economic Development and Cultural Change, University of Chicago Press, vol. 62(2), pages 307-338.
    8. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    9. Nikoloz Kudashvili, 2018. "Sources of Statistical Discrimination: Experimental Evidence from Georgia," CERGE-EI Working Papers wp612, The Center for Economic Research and Graduate Education - Economics Institute, Prague.
    10. Jesse Cunha, 2010. "Testing Paternalism: Cash vs. In-kind Transfer in Rural Mexico," Discussion Papers 09-021, Stanford Institute for Economic Policy Research.
    11. Awotide, Bola Amoke & Awoyemi, Taiwo Timothy & Diagne, Aliou & Ojehomon, Vivian E.T., 2012. "Impact of Seed Voucher System on Rice Farmers’ Welfare in Nigeria: A Randomized Control Trial Approach," 2012 Conference, August 18-24, 2012, Foz do Iguacu, Brazil 124995, International Association of Agricultural Economists.
    12. David K. Evans & Arkadipta Ghosh, 2008. "Prioritizing Educational Investments in Children in the Developing World," Working Papers WR-587, RAND Corporation.
    13. Martin Schlotter & Guido Schwerdt & Ludger Woessmann, 2011. "Econometric methods for causal evaluation of education policies and practices: a non-technical guide," Education Economics, Taylor & Francis Journals, vol. 19(2), pages 109-137.
    14. María laura Alzúa & Guillermo Cruces & Carolina Lopez, 2016. "Long-Run Effects Of Youth Training Programs: Experimental Evidence From Argentina," Economic Inquiry, Western Economic Association International, vol. 54(4), pages 1839-1859, October.
    15. Zifeng Chen & Jiancheng Guan, 2011. "Mapping of biotechnology patents of China from 1995–2008," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(1), pages 73-89, July.
    16. Atonu Rabbani, 2017. "Can Leaders Promote Better Health Behavior? Learning from a Sanitation and Hygiene Communication Experiment in Rural Bangladesh," Working Papers id:11904, eSocialSciences.
    17. Martin, Will, 2021. "Tools for measuring the full impacts of agricultural interventions," IFPRI-MCC technical papers 2, International Food Policy Research Institute (IFPRI).
    18. Deshpande, Ashwini & Desrochers, Alain & Ksoll, Christopher & Shonchoy, Abu S., 2017. "The Impact of a Computer-based Adult Literacy Program on Literacy and Numeracy: Evidence from India," World Development, Elsevier, vol. 96(C), pages 451-473.
    19. Sule Alan & Gyongyi Loranth, 2013. "Subprime Consumer Credit Demand: Evidence from a Lender's Pricing Experiment," The Review of Financial Studies, Society for Financial Studies, vol. 26(9), pages 2353-2374.
    20. Dincecco, Mark & Katz, Gabriel, 2012. "State Capacity and Long-Run Performance," MPRA Paper 38299, University Library of Munich, Germany.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:88:y:2011:i:3:d:10.1007_s11192-011-0440-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.