IDEAS home Printed from https://ideas.repec.org/p/sus/susewp/0122.html
   My bibliography  Save this paper

Confidence Intervals for Recursive Journal Impact Factors

Author

Listed:
  • Johannes Koenig

    (Department of Economics, Economic Policy, Innovation and Entrepreneurship, University of Kassel, Germany)

  • David I. Stern

    (Arndt-Corden Department of Economics, Crawford School of Public Policy, The Australian National University, 132 Lennox Crossing, Acton, ACT 2601, Australia)

  • Richard S.J. Tol

    (Department of Economics, University of Sussex, BN1 9SL Falmer, United Kingdom)

Abstract

We compute confidence intervals for recursive impact factors, that take into account that some citations are more prestigious than others, as well as for the associated ranks of journals, applying the methods to the population of economics journals. The Quarterly Journal of Economics is clearly the journal with greatest impact, the confidence interval for its rank only includes one. Based on the simple bootstrap, the remainder of the "Top-5" journals are in the top 6 together with the Journal of Finance, while the Xie et al. (2009), and Mogstad et al. (2022) methods generally broaden estimated confidence intervals, particularly for mid-ranking journals. All methods agree that most apparent differences in journal quality are, in fact, mostly insignicant.

Suggested Citation

  • Johannes Koenig & David I. Stern & Richard S.J. Tol, 2022. "Confidence Intervals for Recursive Journal Impact Factors," Working Paper Series 0122, Department of Economics, University of Sussex Business School.
    • Handle: RePEc:sus:susewp:0122
    as

    Download full text from publisher

    File URL: https://www.sussex.ac.uk/webteam/gateway/file.php?name=wps-01-2022.pdf&site=18
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Christian Zimmermann, 2013. "Academic Rankings with RePEc," Econometrics, MDPI, vol. 1(3), pages 1-32, December.
    2. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    3. ,, 2014. "A ranking method based on handicaps," Theoretical Economics, Econometric Society, vol. 9(3), September.
    4. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2011. "An updated ranking of academic journals in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 44(4), pages 1525-1538, November.
    5. Kodrzycki Yolanda K. & Yu Pingkang, 2006. "New Approaches to Ranking Economics Journals," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 5(1), pages 1-44, August.
    6. David I. Stern & Richard S. J. Tol, 2021. "Depth and breadth relevance in citation metrics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 961-977, July.
    7. William C. Horrace & Christopher F. Parmeter, 2017. "Accounting for Multiplicity in Inference on Economics Journal Rankings," Southern Economic Journal, John Wiley & Sons, vol. 84(1), pages 337-347, July.
    8. repec:hal:pseose:halshs-01109087 is not listed on IDEAS
    9. Ignacio Palacios-Huerta & Oscar Volij, 2004. "The Measurement of Intellectual Influence," Econometrica, Econometric Society, vol. 72(3), pages 963-977, May.
    10. Kalaitzidakis, P. & Mamuneas, T.P. & Stengos, T., 2003. "Rankings of Academic Journals and Institutions," Working Papers 2003-8, University of Guelph, Department of Economics and Finance.
    11. Andrew J. Oswald, 2007. "An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision‐Makers," Economica, London School of Economics and Political Science, vol. 74(293), pages 21-31, February.
    12. George A. Lozano & Vincent Larivière & Yves Gingras, 2012. "The weakening relationship between the impact factor and papers' citations in the digital age," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(11), pages 2140-2145, November.
    13. Magne Mogstad & Joseph Romano & Azeem Shaikh & Daniel Wilhelm, 2022. "Statistical Uncertainty in the Ranking of Journals and Universities," AEA Papers and Proceedings, American Economic Association, vol. 112, pages 630-634, May.
    14. David I. Stern, 2013. "Uncertainty Measures for Economics Journal Impact Factors," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 173-189, March.
    15. Pantelis Kalaitzidakis & Theofanis P. Mamuneas & Thanasis Stengos, 2003. "Rankings of Academic Journals and Institutions in Economics," Journal of the European Economic Association, MIT Press, vol. 1(6), pages 1346-1366, December.
    16. Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Ryan Chute, 2009. "A Principal Component Analysis of 39 Scientific Impact Measures," PLOS ONE, Public Library of Science, vol. 4(6), pages 1-11, June.
    17. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.
    18. David I Stern, 2014. "High-Ranked Social Science Journal Articles Can Be Identified from Early Citation Information," PLOS ONE, Public Library of Science, vol. 9(11), pages 1-11, November.
    19. Palacios-Huerta, Ignacio & Volij, Oscar, 2014. "Axiomatic measures of intellectual influence," International Journal of Industrial Organization, Elsevier, vol. 34(C), pages 85-90.
    20. George A. Lozano & Vincent Larivière & Yves Gingras, 2012. "The weakening relationship between the impact factor and papers' citations in the digital age," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(11), pages 2140-2145, November.
    21. González-Pereira, Borja & Guerrero-Bote, Vicente P. & Moya-Anegón, Félix, 2010. "A new approach to the metric of journals’ scientific prestige: The SJR indicator," Journal of Informetrics, Elsevier, vol. 4(3), pages 379-391.
    22. Xie, Minge & Singh, Kesar & Zhang, Cun-Hui, 2009. "Confidence Intervals for Population Ranks in the Presence of Ties and Near Ties," Journal of the American Statistical Association, American Statistical Association, vol. 104(486), pages 775-788.
    23. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2010. "Citations versus journal impact factor as proxy of quality: could the latter ever be preferable?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(3), pages 821-833, September.
    24. Per O. Seglen, 1992. "The skewness of science," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 43(9), pages 628-638, October.
    25. Melody Lo & Yong Bao, 2016. "Are Overall Journal Rankings a Good Mapping for Article Quality in Specialty Fields?," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(1), pages 62-67, January.
    26. Johan Lyhagen & Per Ahlgren, 2020. "Uncertainty and the ranking of economics journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2545-2560, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Alexander Butz & Klaus Wohlrabe, 2018. "What are the top five journals in economics? A new meta-ranking," Applied Economics, Taylor & Francis Journals, vol. 50(6), pages 659-675, February.
    2. Ham, John C. & Wright, Julian & Ye, Ziqiu, 2023. "Documenting and Explaining the Dramatic Rise of the New Society Journals in Economics," IZA Discussion Papers 16337, Institute of Labor Economics (IZA).
    3. J. Atsu Amegashie, 2020. "Citations And Incentives In Academic Contests," Economic Inquiry, Western Economic Association International, vol. 58(3), pages 1233-1244, July.
    4. David L. Anderson & John Tressler, 2013. "The Relevance of the “h-” and “g-” Index to Economics in the Context of A Nation-Wide Research Evaluation Scheme: The New Zealand Case," Economic Papers, The Economic Society of Australia, vol. 32(1), pages 81-94, March.
    5. Wohlrabe, Klaus, 2016. "Taking the Temperature: A Meta-Ranking of Economics Journals," MPRA Paper 68933, University Library of Munich, Germany.
    6. Yves Fassin, 2021. "Does the Financial Times FT50 journal list select the best management and economics journals?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5911-5943, July.
    7. David L. Anderson & John Tressler, 2017. "Researcher rank stability across alternative output measurement schemes in the context of a time limited research evaluation: the New Zealand case," Applied Economics, Taylor & Francis Journals, vol. 49(45), pages 4542-4553, September.
    8. Drivas, Kyriakos & Kremmydas, Dimitris, 2020. "The Matthew effect of a journal's ranking," Research Policy, Elsevier, vol. 49(4).
    9. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & J. Martinez-Baena, 2012. "On first quartile journals which are not of highest impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 925-943, March.
    10. John Gibson & David L. Anderson & John Tressler, 2017. "Citations Or Journal Quality: Which Is Rewarded More In The Academic Labor Market?," Economic Inquiry, Western Economic Association International, vol. 55(4), pages 1945-1965, October.
    11. Flores-Szwagrzak, Karol & Treibich, Rafael, 2015. "Co-authorship and the Measurement of Individual Productivity," Discussion Papers on Economics 17/2015, University of Southern Denmark, Department of Economics.
    12. Kosteas, Vasilios D., 2015. "Journal impact factors and month of publication," Economics Letters, Elsevier, vol. 135(C), pages 77-79.
    13. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2011. "Overall prestige of journals with ranking score above a given threshold," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 229-243, October.
    14. Payson Steven, 2019. "Cite This Economics Paper! It Is Time for the House of Cards to Fall Down," Open Economics, De Gruyter, vol. 2(1), pages 1-18, January.
    15. David I. Stern, 2013. "Uncertainty Measures for Economics Journal Impact Factors," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 173-189, March.
    16. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    17. Thelwall, Mike & Fairclough, Ruth, 2015. "Geometric journal impact factors correcting for individual highly cited articles," Journal of Informetrics, Elsevier, vol. 9(2), pages 263-272.
    18. George Emm Halkos & Nickolaos G. Tzeremes, 2011. "Measuring economic journals’ citation efficiency: a data envelopment analysis approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 979-1001, September.
    19. Chan, Ho Fai & Frey, Bruno S. & Gallus, Jana & Torgler, Benno, 2014. "Academic honors and performance," Labour Economics, Elsevier, vol. 31(C), pages 188-204.
    20. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.

    More about this item

    Keywords

    Bibliometrics; citation analysis; publishing; bootstrapping;
    All these keywords.

    JEL classification:

    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C46 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Specific Distributions

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sus:susewp:0122. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: University of Sussex Business School Communications Team (email available below). General contact details of provider: https://edirc.repec.org/data/ecsusuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.