IDEAS home Printed from https://ideas.repec.org/p/knz/dpteco/1002.html
   My bibliography  Save this paper

Research evaluation and journal quality weights: Much ado about nothing?

Author

Listed:
  • Matthias Krapf

    (Department of Economics, University of Konstanz, Germany)

Abstract

Research evaluations based on quality weighted publication output are often criticized on account of the employed journal quality weights. This study shows that evaluations of entire research organizations are very robust with respect to the choice of readily available weighting schemes. We document this robustness by applying rather different weighting schemes to otherwise identical rankings. Our unit of analysis consists of German, Austrian and Swiss university departments in business administration and economics.

Suggested Citation

  • Matthias Krapf, 2010. "Research evaluation and journal quality weights: Much ado about nothing?," Working Paper Series of the Department of Economics, University of Konstanz 2010-02, Department of Economics, University of Konstanz.
    • Handle: RePEc:knz:dpteco:1002
    as

    Download full text from publisher

    File URL: http://www.uni-konstanz.de/FuF/wiwi/workingpaperseries/WP_Krapf-2-10.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. repec:bla:germec:v:9:y:2008:i::p:506-531 is not listed on IDEAS
    2. Friedrich Schneider & Heinrich W. Ursprung, 2008. "The 2008 GEA Journal‐Ranking for the Economics Profession," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 532-538, November.
    3. Fabel Oliver & Hein Miriam & Hofmeister Robert, 2008. "Research Productivity in Business Economics: An Investigation of Austrian, German and Swiss Universities," German Economic Review, De Gruyter, vol. 9(4), pages 506-531, December.
    4. Robert Hofmeister & Heinrich W. Ursprung, 2008. "Das Handelsblatt Ökonomen‐Ranking 2007: Eine kritische Beurteilung," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 254-266, August.
    5. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    6. repec:bla:germec:v:9:y:2008:i::p:402-430 is not listed on IDEAS
    7. Günther G. Schulze & Susanne Warning & Christian Wiermann, 2008. "Zeitschriftenrankings für die Wirtschaftswissenschaften – Konstruktion eines umfassenden Metaindexes," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 286-305, August.
    8. repec:bla:germec:v:9:y:2008:i::p:431-456 is not listed on IDEAS
    9. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    10. Partha, Dasgupta & David, Paul A., 1994. "Toward a new economics of science," Research Policy, Elsevier, vol. 23(5), pages 487-521, September.
    11. repec:jns:jbstat:v:227:y:2007:i:2:p:187-208 is not listed on IDEAS
    12. Pierre-Philippe Combes & Laurent Linnemer, 2003. "Where are the Economists Who Publish? Publication Concentration and Rankings in Europe Based on Cumulative Publications," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1250-1308, December.
    13. Pierre-Philippe Combes & Laurent Linnemer, 2010. "Inferring Missing Citations: A Quantitative Multi-Criteria Ranking of all Journals in Economics," Working Papers halshs-00520325, HAL.
    14. repec:bla:germec:v:9:y:2008:i::p:532-538 is not listed on IDEAS
    15. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    16. repec:bla:germec:v:9:y:2008:i::p:397-401 is not listed on IDEAS
    17. Friedrich Schneider, 2008. "Special Issue on the Economics of the Economics Profession: Editorial," German Economic Review, Verein für Socialpolitik, vol. 9(4), pages 397-401, November.
    18. Ritzberger Klaus, 2008. "A Ranking of Journals in Economics and Related Fields," German Economic Review, De Gruyter, vol. 9(4), pages 402-430, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. J. Paul Elhorst & Katarina Zigova, 2011. "Evidence of Competition in Research Activity among Economic Department using Spatial Econometric Techniques," Working Paper Series of the Department of Economics, University of Konstanz 2011-04, Department of Economics, University of Konstanz.
    2. Tobias Kollmann & Andreas Kuckertz & Christoph Stöckmann & Patrick Krell, 2012. "Die Entrepreneurshipforschung in Deutschland, Österreich und der Schweiz. Eine Resonanzanalyse," ZfKE – Zeitschrift für KMU und Entrepreneurship, Duncker & Humblot, Berlin, vol. 60(1), pages 53-76.
    3. Ioana Alexandra Horodnic & Adriana Zaiţ, 2015. "Motivation and research productivity in a university system undergoing transition," Research Evaluation, Oxford University Press, vol. 24(3), pages 282-292.
    4. Ioana Alexandra HORODNIC, 2014. "Academic Performance: Measurement Methods Used In Socio - Economic Sciences," THE YEARBOOK OF THE "GH. ZANE" INSTITUTE OF ECONOMIC RESEARCHES, Gheorghe Zane Institute for Economic and Social Research ( from THE ROMANIAN ACADEMY, JASSY BRANCH), vol. 23(1), pages 5-17.
    5. Agnes Bäker & Susanne Breuninger & Julia Muschallik & Kerstin Pull & Uschi Backes-Gellner, 2016. "Time to Go? (Inter)National Mobility and Appointment Success of Young Academics," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 17(3), pages 401-421, December.
    6. Kaltrina Nuredini, 2021. "Investigating Altmetric Information For The Top 1000 Journals From Handelsblatt Ranking In Economic And Business Studies," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1315-1343, December.
    7. Matthias Meyer & Rüdiger W. Waldkirch & Michael A. Zaggl, 2012. "Relative Performance Measurement of Researchers: The Impact of Data Source Selection," Schmalenbach Business Review (sbr), LMU Munich School of Management, vol. 64(4), pages 308-330, October.
    8. Sturm Jan-Egbert & Ursprung Heinrich W., 2017. "The Handelsblatt Rankings 2.0: Research Rankings for the Economics Profession in Austria, Germany, and Switzerland," German Economic Review, De Gruyter, vol. 18(4), pages 492-515, December.
    9. Peter Burridge & J. Paul Elhorst & Katarina Zigova, 2016. "Group Interaction in Research and the Use of General Nesting Spatial Models," Advances in Econometrics, in: Spatial Econometrics: Qualitative and Limited Dependent Variables, volume 37, pages 223-258, Emerald Group Publishing Limited.
    10. Guimaraes, Bernardo, 2011. "Qualis as a measuring stick for research output in Economics," Brazilian Review of Econometrics, Sociedade Brasileira de Econometria - SBE, vol. 31(1), March.
    11. Uschi Backes-Gellner & Agnes Bäker & Kerstin Pull, 2018. "The Opportunity Costs of Becoming a Dean: Does Leadership in Academia Crowd Out Research?," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 70(2), pages 189-208, May.
    12. Klaus Gründler & Niklas Potrafke, 2019. "Ideologically-charged terminology: austerity, fiscal consolidation, and sustainable governance," CESifo Working Paper Series 7613, CESifo.
    13. Adam Ayaita & Kerstin Pull & Uschi Backes-Gellner, 2019. "You get what you ‘pay’ for: academic attention, career incentives and changes in publication portfolios of business and economics researchers," Journal of Business Economics, Springer, vol. 89(3), pages 273-290, April.
    14. Julia Muschallik & Kerstin Pull, 2016. "Mentoring in higher education: does it enhance mentees’ research productivity?," Education Economics, Taylor & Francis Journals, vol. 24(2), pages 210-223, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chagas , André Luis Squarize, 2017. "Publish or Perish: um ranking de revistas da subárea de Economia Regional e Urbana para os pesquisadores brasileiros," Revista Brasileira de Estudos Regionais e Urbanos, Associação Brasileira de Estudos Regionais e Urbanos (ABER), vol. 11(4), pages 515-536.
    2. J. Paul Elhorst & Katarina Zigova, 2011. "Evidence of Competition in Research Activity among Economic Department using Spatial Econometric Techniques," Working Paper Series of the Department of Economics, University of Konstanz 2011-04, Department of Economics, University of Konstanz.
    3. Eduardo A. Haddad & Jesus P. Mena-Chalco, Otávio J.G. Sidone, 2016. "Produção Científica e Redes de Colaboração dos Docentes Vinculados aos Programas de Pós-graduação em Economia no Brasil," Working Papers, Department of Economics 2016_10, University of São Paulo (FEA-USP).
    4. Michael Bräuninger & Justus Haucap & Johannes Muck, 2011. "Was lesen und schätzen deutschsprachige Ökonomen heute?," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 12(4), pages 339-371, November.
    5. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    6. George Emm Halkos & Nickolaos G. Tzeremes, 2011. "Measuring economic journals’ citation efficiency: a data envelopment analysis approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 979-1001, September.
    7. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    8. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    9. John P. Conley & Mario J. Crucini & Robert A. Driskill & Ali Sina Onder, 2011. "Incentives and the Effects of Publication Lags on Life Cycle Research Productivity in Economics," Vanderbilt University Department of Economics Working Papers 1122, Vanderbilt University Department of Economics.
    10. John Gibson & David L. Anderson & John Tressler, 2014. "Which Journal Rankings Best Explain Academic Salaries? Evidence From The University Of California," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1322-1340, October.
    11. repec:bla:germec:v:9:y:2008:i::p:457-472 is not listed on IDEAS
    12. Klaus Ritzberger, 2008. "Eine invariante Bewertung wirtschaftswissenschaftlicher Fachzeitschriften," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 9(3), pages 267-285, August.
    13. Koczy, L. & Nichifor, A. & Strobel, M., 2010. "Intellectual Influence: quality versus quantity," Research Memorandum 029, Maastricht University, Maastricht Research School of Economics of Technology and Organization (METEOR).
    14. William C. Horrace & Christopher F. Parmeter, 2017. "Accounting for Multiplicity in Inference on Economics Journal Rankings," Southern Economic Journal, John Wiley & Sons, vol. 84(1), pages 337-347, July.
    15. repec:bla:germec:v:9:y:2008:i::p:402-430 is not listed on IDEAS
    16. Seiler, Christian & Wohlrabe, Klaus, 2012. "Ranking economists on the basis of many indicators: An alternative approach using RePEc data," Journal of Informetrics, Elsevier, vol. 6(3), pages 389-402.
    17. Ham, John C. & Wright, Julian & Ye, Ziqiu, 2023. "Documenting and Explaining the Dramatic Rise of the New Society Journals in Economics," IZA Discussion Papers 16337, Institute of Labor Economics (IZA).
    18. Julio César Arteaga García & Daniel Flores Curiel, 2013. "The Scientific Output of Academic Economists in Mexico from 2000 through 2010," Economía Mexicana NUEVA ÉPOCA, CIDE, División de Economía, vol. 0(1), pages 5-45, January-J.
    19. repec:bla:germec:v:9:y:2008:i::p:473-505 is not listed on IDEAS
    20. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    21. Chan, Ho Fai & Frey, Bruno S. & Gallus, Jana & Torgler, Benno, 2014. "Academic honors and performance," Labour Economics, Elsevier, vol. 31(C), pages 188-204.
    22. László Kóczy & Alexandru Nichifor, 2013. "The intellectual influence of economic journals: quality versus quantity," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 52(3), pages 863-884, April.
    23. Martin Gregor, 2006. "Hodnocení ekonomických pracovišť a ekonomů: Koho, proč, čím a jak [A survey of rankings of economic departments: Global, american, european and national]," Politická ekonomie, Prague University of Economics and Business, vol. 2006(3), pages 394-414.

    More about this item

    Keywords

    Research evaluation; university management;

    JEL classification:

    • I20 - Health, Education, and Welfare - - Education - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:knz:dpteco:1002. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Office Ursprung (email available below). General contact details of provider: https://edirc.repec.org/data/fwkonde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.