IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v9y2015i4p942-953.html
   My bibliography  Save this article

Measuring and comparing the R&D performance of government research institutes: A bottom-up data envelopment analysis approach

Author

Listed:
  • Lee, Seonghee
  • Lee, Hakyeon

Abstract

Government-funded research institutes (GRIs) have played a pivotal role in national R&D in many countries. A prerequisite for achieving desired goals of GRIs with the limited R&D budget is to be able to effectively measure and compare R&D performance of GRIs. This paper proposes the bottom-up approach in which the performance of a GRI is measured based on the efficiency of its R&D projects. Data envelopment analysis (DEA) is employed to measure R&D efficiency of projects, and nonparametric statistical tests are run to measure and compare the R&D performance of GRIs. We apply the bottom-up DEA approach to the performance measurements of 10 Korean GRIs conducting a total of 1481 projects. The two alternatives for incorporating the relative importance of the output variables – the assurance region (AR) model and output integration – are also discussed. The proposed bottom-up approach can be used for formulating and implementing national R&D policy by effectively assessing the performance of GRIs.

Suggested Citation

  • Lee, Seonghee & Lee, Hakyeon, 2015. "Measuring and comparing the R&D performance of government research institutes: A bottom-up data envelopment analysis approach," Journal of Informetrics, Elsevier, vol. 9(4), pages 942-953.
    • Handle: RePEc:eee:infome:v:9:y:2015:i:4:p:942-953
    DOI: 10.1016/j.joi.2015.10.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157715301127
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2015.10.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wang, Eric C. & Huang, Weichiao, 2007. "Relative efficiency of R&D activities: A cross-country study accounting for environmental factors in the DEA approach," Research Policy, Elsevier, vol. 36(2), pages 260-273, March.
    2. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2014. "Assessing national strengths and weaknesses in research fields," Journal of Informetrics, Elsevier, vol. 8(3), pages 766-775.
    3. Kocher, Martin G. & Luptacik, Mikulas & Sutter, Matthias, 2006. "Measuring productivity of research in economics: A cross-country study using DEA," Socio-Economic Planning Sciences, Elsevier, vol. 40(4), pages 314-332, December.
    4. Cherchye, L. & Abeele, P. Vanden, 2005. "On research efficiency: A micro-analysis of Dutch university research in Economics and Business Management," Research Policy, Elsevier, vol. 34(4), pages 495-516, May.
    5. Per Andersen & Niels Christian Petersen, 1993. "A Procedure for Ranking Efficient Units in Data Envelopment Analysis," Management Science, INFORMS, vol. 39(10), pages 1261-1264, October.
    6. Lidia Angulo-Meza & Marcos Lins, 2002. "Review of Methods for Increasing Discrimination in Data Envelopment Analysis," Annals of Operations Research, Springer, vol. 116(1), pages 225-242, October.
    7. Guan, Jiancheng & Chen, Kaihua, 2012. "Modeling the relative efficiency of national innovation systems," Research Policy, Elsevier, vol. 41(1), pages 102-115.
    8. Ortega, José Luis & López-Romero, Elena & Fernández, Inés, 2011. "Multivariate approach to classify research institutes according to their outputs: The case of the CSIC's institutes," Journal of Informetrics, Elsevier, vol. 5(3), pages 323-332.
    9. Eom, Boo-Young & Lee, Keun, 2010. "Determinants of industry-academy linkages and, their impact on firm performance: The case of Korea as a latecomer in knowledge industrialization," Research Policy, Elsevier, vol. 39(5), pages 625-639, June.
    10. Boussofiane, A. & Dyson, R. G. & Thanassoulis, E., 1991. "Applied data envelopment analysis," European Journal of Operational Research, Elsevier, vol. 52(1), pages 1-15, May.
    11. Charnes, A. & Cooper, W. W. & Rhodes, E., 1978. "Measuring the efficiency of decision making units," European Journal of Operational Research, Elsevier, vol. 2(6), pages 429-444, November.
    12. Wei Meng & Zhenhua Hu & Wenbin Liu, 2006. "Efficiency evaluation of basic research in China," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 85-101, October.
    13. R. Allen & A. Athanassopoulos & R.G. Dyson & E. Thanassoulis, 1997. "Weights restrictions and value judgements in Data Envelopment Analysis: Evolution, development and future directions," Annals of Operations Research, Springer, vol. 73(0), pages 13-34, October.
    14. Saaty, Thomas L., 1990. "How to make a decision: The analytic hierarchy process," European Journal of Operational Research, Elsevier, vol. 48(1), pages 9-26, September.
    15. Lee, Hakyeon & Shin, Juneseuk, 2014. "Measuring journal performance for multidisciplinary research: An efficiency perspective," Journal of Informetrics, Elsevier, vol. 8(1), pages 77-88.
    16. Liu, John S. & Lu, Wen-Min, 2010. "DEA and ranking with the network-based approach: a case of R&D performance," Omega, Elsevier, vol. 38(6), pages 453-464, December.
    17. Banker, Rajiv D. & Chang, Hsihui, 2006. "The super-efficiency procedure for outlier identification, not for ranking efficient units," European Journal of Operational Research, Elsevier, vol. 175(2), pages 1311-1320, December.
    18. Seema Sharma & V. J. Thomas, 2008. "Inter-country R&D efficiency analysis: An application of data envelopment analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(3), pages 483-501, September.
    19. Giovanni Abramo & Ciriaco Andrea D'Angelo, 2009. "Assessing technical and cost efficiency of research activities: a case study of the Italian university system," Research Evaluation, Oxford University Press, vol. 18(1), pages 61-70, March.
    20. Johnes, Jill & Johnes, Geraint, 1995. "Research funding and performance in U.K. University Departments of Economics: A frontier analysis," Economics of Education Review, Elsevier, vol. 14(3), pages 301-314, September.
    21. Laurens Cherchye & Willem Moesen & Nicky Rogge & Tom Puyenbroeck, 2007. "An Introduction to ‘Benefit of the Doubt’ Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 82(1), pages 111-145, May.
    22. Groot, Tom & Garcia-Valderrama, Teresa, 2006. "Research quality and efficiency: An analysis of assessments and management issues in Dutch economics and business research programs," Research Policy, Elsevier, vol. 35(9), pages 1362-1376, November.
    23. Hsu, Fang-Ming & Hsueh, Chao-Chih, 2009. "Measuring relative efficiency of government-sponsored R&D projects: A three-stage approach," Evaluation and Program Planning, Elsevier, vol. 32(2), pages 178-186, May.
    24. Adler, Nicole & Friedman, Lea & Sinuany-Stern, Zilla, 2002. "Review of ranking methods in the data envelopment analysis context," European Journal of Operational Research, Elsevier, vol. 140(2), pages 249-265, July.
    25. Tommaso Agasisti & Giuseppe Catalano & Paolo Landoni & Roberto Verganti, 2012. "Evaluating the performance of academic departments: an analysis of research-related output efficiency," Research Evaluation, Oxford University Press, vol. 21(1), pages 2-14, February.
    26. R. D. Banker & A. Charnes & W. W. Cooper, 1984. "Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis," Management Science, INFORMS, vol. 30(9), pages 1078-1092, September.
    27. Thompson, Russell G. & Langemeier, Larry N. & Lee, Chih-Tah & Lee, Euntaik & Thrall, Robert M., 1990. "The role of multiplier bounds in efficiency analysis with application to Kansas farming," Journal of Econometrics, Elsevier, vol. 46(1-2), pages 93-108.
    28. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "A field-standardized application of DEA to national-scale research assessment of universities," Journal of Informetrics, Elsevier, vol. 5(4), pages 618-628.
    29. Andrea Bonaccorsi & Cinzia Daraio, 2003. "A robust nonparametric approach to the analysis of scientific productivity," Research Evaluation, Oxford University Press, vol. 12(1), pages 47-69, April.
    30. Matthew A. Shapiro, 2012. "Receiving information at Korean and Taiwanese universities, industry, and GRIs," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(1), pages 289-309, January.
    31. Lee, Hakyeon & Park, Yongtae & Choi, Hoogon, 2009. "Comparative evaluation of performance of national R&D programs with heterogeneous objectives: A DEA approach," European Journal of Operational Research, Elsevier, vol. 196(3), pages 847-855, August.
    32. William W. Cooper & Lawrence M. Seiford & Kaoru Tone, 2007. "Data Envelopment Analysis," Springer Books, Springer, edition 0, number 978-0-387-45283-8, December.
    33. Meng, Wei & Zhang, Daqun & Qi, Li & Liu, Wenbin, 2008. "Two-level DEA approaches in research evaluation," Omega, Elsevier, vol. 36(6), pages 950-957, December.
    34. Takamura, Yoshiharu & Tone, Kaoru, 2003. "A comparative site evaluation study for relocating Japanese government agencies out of Tokyo," Socio-Economic Planning Sciences, Elsevier, vol. 37(2), pages 85-102, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jung Ho Park & Kwangsoo Shin, 2018. "Efficiency of Government-Sponsored R&D Projects: A Metafrontier DEA Approach," Sustainability, MDPI, vol. 10(7), pages 1-17, July.
    2. Qingqi Long & Ke Song, 2022. "Operational Performance Evaluation of E-government Microblogs Under Emergencies Based on a DEA Method," Information Systems Frontiers, Springer, vol. 24(5), pages 1-18, October.
    3. Xiong, Xi & Yang, Guo-liang & Guan, Zhong-cheng, 2020. "Estimating the multi-period efficiency of high-tech research institutes of the Chinese Academy of Sciences: A dynamic slacks-based measure," Socio-Economic Planning Sciences, Elsevier, vol. 71(C).
    4. Jin-Kyung Kim & Keun-Tae Cho, 2022. "Effects of Technology Commercialization Proactiveness on Commercialization Success: The Case of ETRI in Korea," Sustainability, MDPI, vol. 14(12), pages 1-23, June.
    5. Kun Chen & Xian-tong Ren & Guo-liang Yang & Hai-bo Qin, 2022. "The other side of the coin: The declining of Chinese social science," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 127-143, January.
    6. Xi Xiong & Guo-liang Yang & Kai-di Liu & De-qun Zhou, 2022. "A proposed fixed-sum carryovers reallocation DEA approach for social scientific resources of Chinese public universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4097-4121, July.
    7. Liu, Hui-hui & Yang, Guo-liang & Liu, Xiao-xiao & Song, Yao-yao, 2020. "R&D performance assessment of industrial enterprises in China: A two-stage DEA approach," Socio-Economic Planning Sciences, Elsevier, vol. 71(C).
    8. Prabath Chaminda Abeysiriwardana & Udith K. Jayasinghe-Mudalige, 2022. "Single window performance management: a strategy for evaluation integrated research culture in the commercial agriculture sector," SN Business & Economics, Springer, vol. 2(9), pages 1-24, September.
    9. Junhee Bae & Yanghon Chung & Hyesoo Ko, 2021. "Analysis of efficiency in public research activities in terms of knowledge spillover: focusing on earthquake R&D accomplishments," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 108(2), pages 2249-2264, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lee, Hakyeon & Shin, Juneseuk, 2014. "Measuring journal performance for multidisciplinary research: An efficiency perspective," Journal of Informetrics, Elsevier, vol. 8(1), pages 77-88.
    2. Liu, John S. & Lu, Wen-Min, 2010. "DEA and ranking with the network-based approach: a case of R&D performance," Omega, Elsevier, vol. 38(6), pages 453-464, December.
    3. Mehdi Rhaiem, 2017. "Measurement and determinants of academic research efficiency: a systematic review of the evidence," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 581-615, February.
    4. Jaehun Park & Joonyoung Kim & Si-Il Sung, 2017. "Performance Evaluation of Research and Business Development: A Case Study of Korean Public Organizations," Sustainability, MDPI, vol. 9(12), pages 1-16, December.
    5. Lee, Hakyeon & Park, Yongtae & Choi, Hoogon, 2009. "Comparative evaluation of performance of national R&D programs with heterogeneous objectives: A DEA approach," European Journal of Operational Research, Elsevier, vol. 196(3), pages 847-855, August.
    6. Khoshnevis, Pegah & Teirlinck, Peter, 2018. "Performance evaluation of R&D active firms," Socio-Economic Planning Sciences, Elsevier, vol. 61(C), pages 16-28.
    7. Junhee Bae & Yanghon Chung & Hyesoo Ko, 2021. "Analysis of efficiency in public research activities in terms of knowledge spillover: focusing on earthquake R&D accomplishments," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 108(2), pages 2249-2264, September.
    8. Yang, Guoliang & Ahlgren, Per & Yang, Liying & Rousseau, Ronald & Ding, Jielan, 2016. "Using multi-level frontiers in DEA models to grade countries/territories," Journal of Informetrics, Elsevier, vol. 10(1), pages 238-253.
    9. Jung Ho Park & Kwangsoo Shin, 2018. "Efficiency of Government-Sponsored R&D Projects: A Metafrontier DEA Approach," Sustainability, MDPI, vol. 10(7), pages 1-17, July.
    10. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    11. Kang, Hee Jay & Kim, Changhee & Choi, Kanghwa, 2024. "Combining bootstrap data envelopment analysis with social networks for rank discrimination and suitable potential benchmarks," European Journal of Operational Research, Elsevier, vol. 312(1), pages 283-297.
    12. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    13. Gnewuch, Matthias & Wohlrabe, Klaus, 2018. "Super-efficiency of education institutions: an application to economics departments," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 26, pages 610-623.
    14. Ibrahim Alnafrah, 2021. "Efficiency evaluation of BRICS’s national innovation systems based on bias-corrected network data envelopment analysis," Journal of Innovation and Entrepreneurship, Springer, vol. 10(1), pages 1-28, December.
    15. Esteve, Miriam & Aparicio, Juan & Rodriguez-Sala, Jesus J. & Zhu, Joe, 2023. "Random Forests and the measurement of super-efficiency in the context of Free Disposal Hull," European Journal of Operational Research, Elsevier, vol. 304(2), pages 729-744.
    16. Dariush Khezrimotlagh & Wade D. Cook & Joe Zhu, 2021. "Number of performance measures versus number of decision making units in DEA," Annals of Operations Research, Springer, vol. 303(1), pages 529-562, August.
    17. Kristof De Witte & Laura López-Torres, 2017. "Efficiency in education: a review of literature and a way forward," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 339-363, April.
    18. Ghasemi, Mohammad Reza & Ignatius, Joshua & Rezaee, Babak, 2019. "Improving discriminating power in data envelopment models based on deviation variables framework," European Journal of Operational Research, Elsevier, vol. 278(2), pages 442-447.
    19. Hung, Chia-Liang, 2017. "Social networks, technology ties, and gatekeeper functionality: Implications for the performance management of R&D projects," Research Policy, Elsevier, vol. 46(1), pages 305-315.
    20. Ebrahimi, Bohlool & Dhamotharan, Lalitha & Ghasemi, Mohammad Reza & Charles, Vincent, 2022. "A cross-inefficiency approach based on the deviation variables framework," Omega, Elsevier, vol. 111(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:9:y:2015:i:4:p:942-953. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.