IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v30y2006i4p505-532.html
   My bibliography  Save this article

The Imprecise Science of Evaluating Scholarly Performance

Author

Listed:
  • Thomas Lange

    (Auckland University of Technology)

Abstract

In a growing number of countries, government-appointed assessment panels develop ranks on the basis of the quality of scholarly outputs to apportion budgets in recognition of evaluated performance and to justify public funds for future R&D activities. When business and management journals are being grouped in broad quality categories, a recent study has noted that this procedure was placing the same journals in essentially the same categories. Drawing on journal quality categorizations by several German- and English-speaking business departments and academic associations, the author performs nonparametric tests and correlations to analyze whether this claim can be substantiated. In particular, he examines the ability of broad quality categorizations to add value to governmental, administrative, and academic decision making by withstanding the criticism traditionally levied at research quality assessments.

Suggested Citation

  • Thomas Lange, 2006. "The Imprecise Science of Evaluating Scholarly Performance," Evaluation Review, , vol. 30(4), pages 505-532, August.
    • Handle: RePEc:sae:evarev:v:30:y:2006:i:4:p:505-532
    DOI: 10.1177/0193841X05284088
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X05284088
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X05284088?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Lyall, Catherine & Bruce, Ann & Firn, John & Firn, Marion & Tait, Joyce, 2004. "Assessing end-use relevance of public sector research organisations," Research Policy, Elsevier, vol. 33(1), pages 73-87, January.
    2. Apostolos Ballas & Vasilis Theoharakis, 2003. "Exploring Diversity in Accounting through Faculty Journal Perceptions," Contemporary Accounting Research, John Wiley & Sons, vol. 20(4), pages 619-644, December.
    3. Henderson, Glenn Jr. & Ganesh, Gopala K. & Chandy, P. R., 1990. "Across-discipline journal awareness and evaluation: Implications for the promotion and tenure process," Journal of Economics and Business, Elsevier, vol. 42(4), pages 325-351, November.
    4. Michael H. MacRoberts & Barbara R. MacRoberts, 1989. "Problems of citation analysis: A critical review," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 40(5), pages 342-349, September.
    5. Anne Loft & Ann Jorissen & Peter Walton, 2002. "From newsletter to academic journal: creating the European Accounting Review," European Accounting Review, Taylor & Francis Journals, vol. 11(1), pages 43-75.
    6. HOLMSTROM, Bengt, 1979. "Moral hazard and observability," LIDAM Reprints CORE 379, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    7. Vastag, Gyula & Montabon, Frank, 2002. "Journal characteristics, rankings and social acculturation in operations management," Omega, Elsevier, vol. 30(2), pages 109-126, April.
    8. Clive Beed & Cara Beed, 1996. "Measuring the Quality of Academic Journals: The Case of Economics," Journal of Post Keynesian Economics, Taylor & Francis Journals, vol. 18(3), pages 369-396, March.
    9. Quirin Schiermeier, 2004. "Grade expectations for German research institutes," Nature, Nature, vol. 432(7015), pages 260-260, November.
    10. Cudd, Mike & Morris, Joe, 1988. "Bias in Journal Ratings," The Financial Review, Eastern Finance Association, vol. 23(1), pages 117-125, February.
    11. Alexander, John C & Mabry, Rodney H, 1994. "Relative Significance of Journals, Authors, and Articles Cited in Financial Research," Journal of Finance, American Finance Association, vol. 49(2), pages 697-712, June.
    12. Per O. Seglen, 1994. "Causal relationship between article citedness and journal impact," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 45(1), pages 1-11, January.
    13. Brown, Lawrence D, 2003. "Ranking Journals Using Social Science Research Network Downloads," Review of Quantitative Finance and Accounting, Springer, vol. 20(3), pages 291-307, May.
    14. Kam C. Chan & Carl R. Chen & Thomas L. Steiner, 2002. "Production in the Finance Literature, Institutional Reputation, and Labor Mobility in Academia: A Global Perspective," Financial Management, Financial Management Association, vol. 31(4), Winter.
    15. Bosch, Darrell, 2002. "Organisation for Economic Co-operation and Development, Income Risk Management in Agriculture: OECD, Paris, France, December 2000, 152 pp., US$ 42.00, Softcover, ISBN 92-64-18534-8," Agricultural Economics, Blackwell, vol. 27(1), pages 88-90, May.
    16. Lawrence D. Brown & Ronald J. Huefner, 1994. "The Familiarity with and Perceived Quality of Accounting Journals: Views of Senior Accounting Faculty in Leading U.S. MBA Programs," Contemporary Accounting Research, John Wiley & Sons, vol. 11(1), pages 223-250, June.
    17. Bengt Holmstrom, 1979. "Moral Hazard and Observability," Bell Journal of Economics, The RAND Corporation, vol. 10(1), pages 74-91, Spring.
    18. Lukka, Kari & Kasanen, Eero, 1996. "Is accounting a global or a local discipline? evidence from major research journals," Accounting, Organizations and Society, Elsevier, vol. 21(7-8), pages 755-773.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Walters, William H., 2017. "Do subjective journal ratings represent whole journals or typical articles? Unweighted or weighted citation impact?," Journal of Informetrics, Elsevier, vol. 11(3), pages 730-744.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Edward P. Swanson, 2004. "Publishing in the Majors: A Comparison of Accounting, Finance, Management, and Marketing," Contemporary Accounting Research, John Wiley & Sons, vol. 21(1), pages 223-255, March.
    2. Bernard Raffournier & Alain Schatt, 2010. "Is European Accounting Research Fairly Reflected in Academic Journals? An Investigation of Possible Non-mainstream and Language Barrier Biases," European Accounting Review, Taylor & Francis Journals, vol. 19(1), pages 161-190.
    3. Domingo Docampo & Vicente Safón, 2021. "Journal ratings: a paper affiliation methodology," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8063-8090, September.
    4. Currie, Russell R. & Pandher, Gurupdesh S., 2011. "Finance journal rankings and tiers: An Active Scholar Assessment methodology," Journal of Banking & Finance, Elsevier, vol. 35(1), pages 7-20, January.
    5. Wade D. Cook & Tal Raviv & Alan J. Richardson, 2010. "Aggregating Incomplete Lists of Journal Rankings: An Application to Academic Accounting Journals," Accounting Perspectives, John Wiley & Sons, vol. 9(3), pages 217-235, September.
    6. Hussain, Simon, 2010. "Accounting journals and the ABS quality ratings," The British Accounting Review, Elsevier, vol. 42(1), pages 1-16.
    7. Lowe, Alan & Locke, Joanne, 2005. "Perceptions of journal quality and research paradigm: results of a web-based survey of British accounting academics," Accounting, Organizations and Society, Elsevier, vol. 30(1), pages 81-98, January.
    8. Geert Campenhout & Tom Caneghem & Steve Uytbergen, 2008. "A comparison of overall and sub-area journal influence: The case of the accounting literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 77(1), pages 61-90, October.
    9. Alan Goodacre & Clive Gaunt & Darren Henry, 2021. "Publication records of Australian accounting and finance faculty promoted to full professor, set within an international context," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 61(2), pages 3089-3133, June.
    10. Groot, Tom & Garcia-Valderrama, Teresa, 2006. "Research quality and efficiency: An analysis of assessments and management issues in Dutch economics and business research programs," Research Policy, Elsevier, vol. 35(9), pages 1362-1376, November.
    11. Apostolos Ballas & Vasilis Theoharakis, 2003. "Exploring Diversity in Accounting through Faculty Journal Perceptions," Contemporary Accounting Research, John Wiley & Sons, vol. 20(4), pages 619-644, December.
    12. Lowe, Alan & Locke, Joanne, 2006. "Constructing an ‘efficient frontier’ of accounting journal quality," The British Accounting Review, Elsevier, vol. 38(3), pages 321-341.
    13. Chan, Kam C. & Chan, Kam C. & Seow, Gim S. & Tam, Kinsun, 2009. "Ranking accounting journals using dissertation citation analysis: A research note," Accounting, Organizations and Society, Elsevier, vol. 34(6-7), pages 875-885, August.
    14. Merchant, Kenneth A. & Van der Stede, Wim A. & Zheng, Liu, 2003. "Disciplinary constraints on the advancement of knowledge: the case of organizational incentive systems," Accounting, Organizations and Society, Elsevier, vol. 28(2-3), pages 251-286.
    15. Vivien Beattie & Alan Goodacre, 2006. "A new method for ranking academic journals in accounting and finance," Accounting and Business Research, Taylor & Francis Journals, vol. 36(2), pages 65-91.
    16. Calcagno, R. & Renneboog, L.D.R., 2004. "Capital Structure and Managerial Compensation : The Effects of Renumeration Seniority," Discussion Paper 2004-120, Tilburg University, Center for Economic Research.
    17. Yaofeng Fu & Ruokun Huang & Yiran Sheng, 2017. "Labor Contract Law -An Economic View," Papers 1702.03977, arXiv.org.
    18. McCausland, David & Pouliakas, Konstantinos & Theodossiou, Ioannis, 2005. "Some are Punished and Some are Rewarded: A Study of the Impact of Performance Pay on Job Satisfaction," MPRA Paper 14243, University Library of Munich, Germany.
    19. Rajesh K. Aggarwal & Andrew A. Samwick, 1999. "Executive Compensation, Strategic Competition, and Relative Performance Evaluation: Theory and Evidence," Journal of Finance, American Finance Association, vol. 54(6), pages 1999-2043, December.
    20. David Martimort & Flavio Menezes & Myrna Wooders & ELISABETTA IOSSA & DAVID MARTIMORT, 2015. "The Simple Microeconomics of Public-Private Partnerships," Journal of Public Economic Theory, Association for Public Economic Theory, vol. 17(1), pages 4-48, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:30:y:2006:i:4:p:505-532. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.