IDEAS home Printed from https://ideas.repec.org/p/aeg/report/2017-04.html
   My bibliography  Save this paper

A framework for the Assessment of Research and its impacts

Author

Listed:
  • Cinzia Daraio

    (Department of Computer, Control and Management Engineering Antonio Ruberti (DIAG), University of Rome La Sapienza, Rome, Italy)

Abstract

This paper proposes a framework for the development of models for the assessment of research activities and their impacts. It distinguishes three dimensions: theory, methodology and data, each of which is further characterized by three main building blocks: education, research and innovation (theory); efficiency, effectiveness and impact (methodology); and availability, interoperability and \unit free" property (data). The different dimensions and their nine constituent building blocks are attributes of an overarching concept, denoted as "quality". Three additional quality attributes are identified as implementation factors (tailorability, transparency and openness) and three "enabling" conditions (convergence, mixed methods and knowledge infrastructures) complete the framework.The paper illustrates the complexity of the evaluation describing the generalized "implementation problem" in research assessment, according to the proposed framework. A framework is required to develop models of metrics. Models of metrics are necessary to assess the meaning, validity and robustness of metrics. The proposed framework can be a useful reference for the development of the ethics of research evaluation. Three examples of application, as well as further directions for future research are provided.

Suggested Citation

  • Cinzia Daraio, 2017. "A framework for the Assessment of Research and its impacts," DIAG Technical Reports 2017-04, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    • Handle: RePEc:aeg:report:2017-04
    as

    Download full text from publisher

    File URL: http://wwwold.dis.uniroma1.it/~bibdis/index.php?option=com_docman&task=doc_download&gid=116&Itemid=34
    File Function: First version, 2017
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Romer, Paul M, 1986. "Increasing Returns and Long-run Growth," Journal of Political Economy, University of Chicago Press, vol. 94(5), pages 1002-1037, October.
    2. Etzkowitz, Henry & Leydesdorff, Loet, 2000. "The dynamics of innovation: from National Systems and "Mode 2" to a Triple Helix of university-industry-government relations," Research Policy, Elsevier, vol. 29(2), pages 109-123, February.
    3. Chien-Ming Chen & Magali A. Delmas & Marvin B. Lieberman, 2015. "Production frontier methodologies and efficiency as a performance measure in strategic management research," Strategic Management Journal, Wiley Blackwell, vol. 36(1), pages 19-36, January.
    4. Oecd, 2015. "Making Open Science a Reality," OECD Science, Technology and Industry Policy Papers 25, OECD Publishing.
    5. Hanushek, Eric A. & Woessmann, Ludger, 2007. "The role of education quality for economic growth," Policy Research Working Paper Series 4122, The World Bank.
    6. Hanushek, Eric A. & Jamison, Dean T. & Jamison, Eliot A. & Wößmann, Ludger, 2008. "Education and economic growth: It’s not just going to school, but learning something while there that matters," Munich Reprints in Economics 20467, University of Munich, Department of Economics.
    7. Cinzia Daraio & Maurizio Lenzerini & Claudio Leporelli & Henk F. Moed & Paolo Naggar & Andrea Bonaccorsi & Alessandro Bartolucci, 2016. "Data integration for research and innovation policy: an Ontology-Based Data Management approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 857-871, February.
    8. Wolfgang Glänzel & Henk F. Moed, 2013. "Opinion paper: thoughts and facts on bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 381-394, July.
    9. Leibenstein, Harvey, 1978. "On the Basic Proposition of X-Efficiency Theory," American Economic Review, American Economic Association, vol. 68(2), pages 328-332, May.
    10. Zvi Griliches, 1998. "Productivity, R&D, and the Data Constraint," NBER Chapters, in: R&D and Productivity: The Econometric Evidence, pages 347-374, National Bureau of Economic Research, Inc.
    11. Lane, Julia I. & Owen-Smith, Jason & Rosen, Rebecca F. & Weinberg, Bruce A., 2015. "New linked data on research investments: Scientific workforce, productivity, and public value," Research Policy, Elsevier, vol. 44(9), pages 1659-1671.
    12. Hamid Ekbia & Michael Mattioli & Inna Kouper & G. Arave & Ali Ghazinejad & Timothy Bowman & Venkata Ratandeep Suri & Andrew Tsou & Scott Weingart & Cassidy R. Sugimoto, 2015. "Big data, bigger dilemmas: A critical review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(8), pages 1523-1545, August.
    13. Léopold Simar & Paul W. Wilson, 2015. "Statistical Approaches for Non-parametric Frontier Models: A Guided Tour," International Statistical Review, International Statistical Institute, vol. 83(1), pages 77-110, April.
    14. Mansfield, Edwin, 1995. "Academic Research Underlying Industrial Innovations:," The Review of Economics and Statistics, MIT Press, vol. 77(1), pages 55-65, February.
    15. Stewart, Jenny, 1995. "Models of priority-setting for public sector research," Research Policy, Elsevier, vol. 24(1), pages 115-126, January.
    16. Salter, Ammon J. & Martin, Ben R., 2001. "The economic benefits of publicly funded basic research: a critical review," Research Policy, Elsevier, vol. 30(3), pages 509-532, March.
    17. Shahidur R. Khandker & Gayatri B. Koolwal & Hussain A. Samad, 2010. "Handbook on Impact Evaluation : Quantitative Methods and Practices," World Bank Publications - Books, The World Bank Group, number 2693.
    18. Rosenberg, Nathan & Nelson, Richard R., 1994. "American universities and technical advance in industry," Research Policy, Elsevier, vol. 23(3), pages 323-348, May.
    19. Jorge Galán & Helena Veiga & Michael Wiper, 2014. "Bayesian estimation of inefficiency heterogeneity in stochastic frontier models," Journal of Productivity Analysis, Springer, vol. 42(1), pages 85-101, August.
    20. Leibenstein, Harvey, 1978. "General X-Efficiency Theory and Economic Development," OUP Catalogue, Oxford University Press, number 9780195023800.
    21. Martin, Ben R., 2012. "The evolution of science policy and innovation studies," Research Policy, Elsevier, vol. 41(7), pages 1219-1239.
    22. Mansfield, Edwin, 1991. "Academic research and industrial innovation," Research Policy, Elsevier, vol. 20(1), pages 1-12, February.
    23. Golan, Amos, 2008. "Information and Entropy Econometrics — A Review and Synthesis," Foundations and Trends(R) in Econometrics, now publishers, vol. 2(1–2), pages 1-145, February.
    24. Perkmann, Markus & Tartari, Valentina & McKelvey, Maureen & Autio, Erkko & Broström, Anders & D’Este, Pablo & Fini, Riccardo & Geuna, Aldo & Grimaldi, Rosa & Hughes, Alan & Krabel, Stefan & Kitson, Mi, 2013. "Academic engagement and commercialisation: A review of the literature on university–industry relations," Research Policy, Elsevier, vol. 42(2), pages 423-442.
    25. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Rankings and university performance: A conditional multidimensional approach," European Journal of Operational Research, Elsevier, vol. 244(3), pages 918-930.
    26. Herbert Simon, 2000. "Bounded rationality in social science: Today and tomorrow," Mind & Society: Cognitive Studies in Economics and Social Sciences, Springer;Fondazione Rosselli, vol. 1(1), pages 25-39, March.
    27. Daraio, Cinzia & Simar, Léopold, 2014. "Directional distances and their robust versions: Computational and testing issues," European Journal of Operational Research, Elsevier, vol. 237(1), pages 358-369.
    28. Marcus R. Munafò & Brian A. Nosek & Dorothy V. M. Bishop & Katherine S. Button & Christopher D. Chambers & Nathalie Percie du Sert & Uri Simonsohn & Eric-Jan Wagenmakers & Jennifer J. Ware & John P. A, 2017. "A manifesto for reproducible science," Nature Human Behaviour, Nature, vol. 1(1), pages 1-9, January.
    29. Christopher F. Parmeter & Alan T. K. Wan & Xinyu Zhang, 2019. "Model averaging estimators for the stochastic frontier model," Journal of Productivity Analysis, Springer, vol. 51(2), pages 91-103, June.
    30. Tony Lawson, 2012. "Ontology and the study of social reality: emergence, organisation, community, power, social relations, corporations, artefacts and money," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 36(2), pages 345-385.
    31. Blaise Cronin, 2013. "Thinking about data," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(3), pages 435-436, March.
    32. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    33. Daraio, Cinzia & Simar, Leopold & Wilson, Paul, 2016. "Nonparametric Estimation of Efficiency in the Presence of Environmental Variables," LIDAM Discussion Papers ISBA 2016027, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    34. Richard Owen & Phil Macnaghten & Jack Stilgoe, 2012. "Responsible research and innovation: From science in society to science for society, with society," Science and Public Policy, Oxford University Press, vol. 39(6), pages 751-760, December.
    35. Zvi Griliches, 1998. "R&D and Productivity: The Econometric Evidence," NBER Books, National Bureau of Economic Research, Inc, number gril98-1, June.
    36. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    37. Borrás, Susana & Edquist, Charles, 2013. "The choice of innovation policy instruments," Technological Forecasting and Social Change, Elsevier, vol. 80(8), pages 1513-1522.
    38. Ki-Seok Kwon & Han Woo Park & Minho So & Loet Leydesdorff, 2012. "Has globalization strengthened South Korea’s national research system? National and international dynamics of the Triple Helix of scientific co-authorship relationships in South Korea," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(1), pages 163-176, January.
    39. Griliches, Zvi, 1986. "Economic data issues," Handbook of Econometrics, in: Z. Griliches† & M. D. Intriligator (ed.), Handbook of Econometrics, edition 1, volume 3, chapter 25, pages 1465-1514, Elsevier.
    40. Mark A. Largent & Julia I. Lane, 2012. "STAR METRICS and the Science of Science Policy," Review of Policy Research, Policy Studies Organization, vol. 29(3), pages 431-438, May.
    41. Ludger Vessman & Jerik Hanushek, 2007. "The role of education quality in economic growth (Part I)," Voprosy obrazovaniya / Educational Studies Moscow, National Research University Higher School of Economics, issue 2, pages 86-116.
    42. Fagerberg, Jan & Martin, Ben R. & Andersen, Esben Sloth (ed.), 2013. "Innovation Studies: Evolution and Future Challenges," OUP Catalogue, Oxford University Press, number 9780199686353.
    43. Alex Graves & Greg Wayne & Malcolm Reynolds & Tim Harley & Ivo Danihelka & Agnieszka Grabska-Barwińska & Sergio Gómez Colmenarejo & Edward Grefenstette & Tiago Ramalho & John Agapiou & Adrià Puigdomèn, 2016. "Hybrid computing using a neural network with dynamic external memory," Nature, Nature, vol. 538(7626), pages 471-476, October.
    44. Bronwyn H. Hall & Nathan Rosenberg (ed.), 2010. "Handbook of the Economics of Innovation," Handbook of the Economics of Innovation, Elsevier, edition 1, volume 1, number 1.
    45. Saba Hinrichs-Krapels & Jonathan Grant, 2016. "Exploring the effectiveness, efficiency and equity (3e’s) of research and research impact assessment," Palgrave Communications, Palgrave Macmillan, vol. 2(1), pages 1-9, December.
    46. Checchi,Daniele, 2008. "The Economics of Education," Cambridge Books, Cambridge University Press, number 9780521066464, November.
    47. Jérôme Vandenbussche & Philippe Aghion & Costas Meghir, 2006. "Growth, distance to frontier and composition of human capital," Journal of Economic Growth, Springer, vol. 11(2), pages 97-127, June.
    48. Geraint Johnes & Jill Johnes (ed.), 2004. "International Handbook on the Economics of Education," Books, Edward Elgar Publishing, number 2847.
    49. Peter Dahler-Larsen, 2014. "Constitutive Effects of Performance Indicators: Getting beyond unintended consequences," Public Management Review, Taylor & Francis Journals, vol. 16(7), pages 969-986, October.
    50. Feldman, Maryann & Kenney, Martin & Lissoni, Francesco, 2015. "The New Data Frontier," Research Policy, Elsevier, vol. 44(9), pages 1629-1632.
    51. Fredrik Niclas Piro & Gunnar Sivertsen, 2016. "How can differences in international university rankings be explained?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2263-2278, December.
    52. Martin Kowarsch & Jennifer Garard & Pauline Riousset & Dominic Lenzi & Marcel J. Dorsch & Brigitte Knopf & Jan-Albrecht Harrs & Ottmar Edenhofer, 2016. "Scientific assessments to facilitate deliberative policy learning," Palgrave Communications, Palgrave Macmillan, vol. 2(1), pages 1-20, December.
    53. Hildebrandt, Steen, 1977. "Implementation of the operations research/management science process," European Journal of Operational Research, Elsevier, vol. 1(5), pages 289-294, September.
    54. Harvey Leibenstein, 1975. "Aspects of the X-Efficiency Theory of the Firm," Bell Journal of Economics, The RAND Corporation, vol. 6(2), pages 580-606, Autumn.
    55. Rebora, Gianfranco & Turri, Matteo, 2013. "The UK and Italian research assessment exercises face to face," Research Policy, Elsevier, vol. 42(9), pages 1657-1666.
    56. Leibenstein, Harvey & Maital, Shlomo, 1992. "Empirical Estimation and Partitioning of X-Inefficiency: A Data-Envelopment Approach," American Economic Review, American Economic Association, vol. 82(2), pages 428-433, May.
    57. Griliches, Zvi, 1998. "R&D and Productivity," National Bureau of Economic Research Books, University of Chicago Press, edition 1, number 9780226308869, October.
    58. Steven Hill, 2016. "Assessing (for) impact: future assessment of the societal impact of research," Palgrave Communications, Palgrave Macmillan, vol. 2(1), pages 1-7, December.
    59. Blaise Cronin, 2013. "Thinking about data," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(3), pages 435-436, March.
    60. Mirowski, Philip & Sent, Esther-Mirjam (ed.), 2002. "Science Bought and Sold," University of Chicago Press Economics Books, University of Chicago Press, edition 1, number 9780226538563, October.
    61. Paul M. Romer, 1994. "The Origins of Endogenous Growth," Journal of Economic Perspectives, American Economic Association, vol. 8(1), pages 3-22, Winter.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cinzia Daraio & Leopold Simar & Paul W. Wilson, 2019. "Quality and its Impact on Efficiency," LEM Papers Series 2019/06, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    2. Cinzia Daraio & Alessio Vaccari, 2019. "Sorting out Guidelines for a Good Evaluation of Research Practices.Towards the Assessment of Researcher’s Virtues," DIAG Technical Reports 2019-10, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    3. Cinzia Daraio & Simone Leo & Monica Scannapieco, 2022. "Accounting for quality in data integration systems: a completeness-aware integration approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(3), pages 1465-1490, March.
    4. Daraio, Cinzia & Simar, Léopold & Wilson, Paul W., 2021. "Quality as a latent heterogeneity factor in the efficiency of universities," Economic Modelling, Elsevier, vol. 99(C).
    5. Cinzia Daraio & Alessio Vaccari, 2019. "Sorting out Guidelines for the Good Evaluation of Research Practices," DIAG Technical Reports 2019-02, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    6. Cinzia Daraio & Alessio Vaccari, 2020. "Using normative ethics for building a good evaluation of research practices: towards the assessment of researcher’s virtues," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1053-1075, November.
    7. Marco Angelini & Cinzia Daraio & Maurizio Lenzerini & Francesco Leotta & Giuseppe Santucci, 2020. "Performance model’s development: a novel approach encompassing ontology-based data access and visual analytics," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 865-892, November.
    8. Marco Angelini & Cinzia Daraio & Maurizio Lenzerini & Francesco Leotta & Giuseppe Santucci, 2019. "Performance Model’s development: A Novel Approach encompassing Ontology-Based Data Access and Visual Analytics," DIAG Technical Reports 2019-11, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".
    9. Moriah B. Bostian & Cinzia Daraio & Rolf Fare & Shawna Grosskopf & Maria Grazia Izzo & Luca Leuzzi & Giancarlo Ruocco & William L. Weber, 2018. "Inference for Nonparametric Productivity Networks: A Pseudo-likelihood Approach," DIAG Technical Reports 2018-06, Department of Computer, Control and Management Engineering, Universita' degli Studi di Roma "La Sapienza".

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Van Elk, Roel & Verspagen, Bart & Ter Weel, Bas & Van der Wiel, Karen & Wouterse, Bram, 2015. "A macroeconomic analysis of the returns to public R&D investments," MERIT Working Papers 2015-042, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    2. Van Elk, Roel & Verspagen, Bart & Ter Weel, Bas & Van der Wiel, Karen & Wouterse, Bram, 2015. "A macroeconomic analysis of the returns to public R&D investments," MERIT Working Papers 2015-042, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).
    3. Igors Skute & Kasia Zalewska-Kurek & Isabella Hatak & Petra Weerd-Nederhof, 2019. "Mapping the field: a bibliometric analysis of the literature on university–industry collaborations," The Journal of Technology Transfer, Springer, vol. 44(3), pages 916-947, June.
    4. Ugur, Mehmet & Trushin, Eshref & Solomon, Edna & Guidi, Francesco, 2016. "R&D and productivity in OECD firms and industries: A hierarchical meta-regression analysis," Research Policy, Elsevier, vol. 45(10), pages 2069-2086.
    5. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2014. "Assessing national strengths and weaknesses in research fields," Journal of Informetrics, Elsevier, vol. 8(3), pages 766-775.
    6. Ugo Rizzo & Nicolò Barbieri & Laura Ramaciotti & Demian Iannantuono, 2020. "The division of labour between academia and industry for the generation of radical inventions," The Journal of Technology Transfer, Springer, vol. 45(2), pages 393-413, April.
    7. Albahari, Alberto & Pérez-Canto, Salvador & Barge-Gil, Andrés & Modrego, Aurelia, 2017. "Technology Parks versus Science Parks: Does the university make the difference?," Technological Forecasting and Social Change, Elsevier, vol. 116(C), pages 13-28.
    8. A. Bellucci & L. Pennacchio, 2016. "University knowledge and firm innovation: evidence from European countries," The Journal of Technology Transfer, Springer, vol. 41(4), pages 730-752, August.
    9. Gersbach, Hans & Schneider, Maik & Schetter, Ulrich, 2015. "How Much Science? The 5 Ws (and 1 H) of Investing in Basic Research," CEPR Discussion Papers 10482, C.E.P.R. Discussion Papers.
    10. Roel van Elk & Bas ter Weel & Karen van der Wiel & Bram Wouterse, 2019. "Estimating the Returns to Public R&D Investments: Evidence from Production Function Models," De Economist, Springer, vol. 167(1), pages 45-87, March.
    11. Mario Coccia, 2006. "Economic and social studies of scientific research: nature and origins," CERIS Working Paper 200607, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.
    12. Daraio, Cinzia & Moed, Henk F., 2011. "Is Italian science declining?," Research Policy, Elsevier, vol. 40(10), pages 1380-1392.
    13. Camilla Mastromarco & Léopold Simar, 2021. "Latent heterogeneity to evaluate the effect of human capital on world technology frontier," Journal of Productivity Analysis, Springer, vol. 55(2), pages 71-89, April.
    14. Stavins, Robert & Jaffe, Adam & Newell, Richard, 2000. "Technological Change and the Environment," Working Paper Series rwp00-002, Harvard University, John F. Kennedy School of Government.
    15. Martina Kauffeld-Monz, 2005. "Knowledge spillovers within regional networks of innovation and the contribution made by public research," ERSA conference papers ersa05p440, European Regional Science Association.
    16. Hottenrott, Hanna & Thorwarth, Susanne, 2010. "Industry funding of university research and scientific productivity," ZEW Discussion Papers 10-105, ZEW - Leibniz Centre for European Economic Research.
    17. Joern Block & Roy Thurik & Haibo Zhou, 2013. "What turns knowledge into innovative products? The role of entrepreneurship and knowledge spillovers," Journal of Evolutionary Economics, Springer, vol. 23(4), pages 693-718, September.
    18. Perkmann, Markus & King, Zella & Pavelin, Stephen, 2011. "Engaging excellence? Effects of faculty quality on university engagement with industry," Research Policy, Elsevier, vol. 40(4), pages 539-552, May.
    19. Stefano Bianchini & Francesco Lissoni & Michele Pezzoni & Lorenzo Zirulia, 2016. "The economics of research, consulting, and teaching quality: theory and evidence from a technical university," Economics of Innovation and New Technology, Taylor & Francis Journals, vol. 25(7), pages 668-691, October.
    20. Schubert, Torben & Jäger, Angela & Türkeli, Serdar & Visentin, Fabiana, 2020. "Addressing the productivity paradox with big data: A literature review and adaptation of the CDM econometric model," MERIT Working Papers 2020-050, United Nations University - Maastricht Economic and Social Research Institute on Innovation and Technology (MERIT).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aeg:report:2017-04. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Antonietta Angelica Zucconi (email available below). General contact details of provider: https://edirc.repec.org/data/dirosit.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.