IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v79y2009i2d10.1007_s11192-009-0420-4.html
   My bibliography  Save this article

Aggregation properties of relative impact and other classical indicators: Convexity issues and the Yule-Simpson paradox

Author

Listed:
  • Suzy Ramanana-Rahary

    (Observatoire des Sciences et des Techniques (OST))

  • Michel Zitt

    (Observatoire des Sciences et des Techniques (OST)
    INRA-Lereco)

  • Ronald Rousseau

    (KHBO — Industrial Sciences and Technology
    K.U. Leuven)

Abstract

Among classical bibliometric indicators, direct and relative impact measures for countries or other players in science are appealing and standard. Yet, as shown in this article, they may exhibit undesirable statistical properties, or at least ones that pose questions of interpretation in evaluation and benchmarking contexts. In this article, we address two such properties namely sensitivity to the Yule-Simpson effect, and a problem related to convexity. The Yule-Simpson effect can occur for direct impacts and, in a variant form, for relative impact, causing an apparent incoherence between field values and the aggregate (all-fields) value. For relative impacts, it may result in a severe form of ‘out-range’ of aggregate values, where a player’s relative impact shifts from ‘good’ to ‘bad’, or conversely. Out-range and lack of convexity in general are typical of relative impact indicators. Using empirical data, we suggest that, for relative impact measures, ‘out-range’ due to lack of convexity is not exceptional. The Yule-Simpson effect is less frequent, and especially occurs for small players with particular specialisation profiles.

Suggested Citation

  • Suzy Ramanana-Rahary & Michel Zitt & Ronald Rousseau, 2009. "Aggregation properties of relative impact and other classical indicators: Convexity issues and the Yule-Simpson paradox," Scientometrics, Springer;Akadémiai Kiadó, vol. 79(2), pages 311-327, May.
    • Handle: RePEc:spr:scient:v:79:y:2009:i:2:d:10.1007_s11192-009-0420-4
    DOI: 10.1007/s11192-009-0420-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-009-0420-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-009-0420-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hariolf Grupp, 1998. "Foundations of the Economics of Innovation," Books, Edward Elgar Publishing, number 1390.
    2. Michel Zitt & Suzy Ramanana-Rahary & Elise Bassecoulard, 2003. "Correcting glasses help fair comparisons in international science landscape: Country indicators as a function of ISI database delineation," Scientometrics, Springer;Akadémiai Kiadó, vol. 56(2), pages 259-282, February.
    3. Michel Zitt & Elise Bassecoulard & Yoshiko Okubo, 2000. "Shadows of the Past in International Cooperation: Collaboration Profiles of the Top Five Producers of Science," Scientometrics, Springer;Akadémiai Kiadó, vol. 47(3), pages 627-657, March.
    4. Poovanalingam Murugesan & Michael J. Moravcsik, 1978. "Variation of the nature of citation measures with journals and scientific specialties," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 29(3), pages 141-147, May.
    5. Michel Zitt & Suzy Ramanana-Rahary & Elise Bassecoulard, 2005. "Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(2), pages 373-401, April.
    6. Péter Vinkler, 2003. "Relations of relative scientometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 58(3), pages 687-694, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. M. Zitt, 2011. "Behind citing-side normalization of citations: some properties of the journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 329-344, October.
    2. Rousseau, Ronald & Yang, Liying, 2012. "Reflections on the activity index and related indicators," Journal of Informetrics, Elsevier, vol. 6(3), pages 413-421.
    3. Michel Zitt, 2012. "The journal impact factor: angel, devil, or scapegoat? A comment on J.K. Vanclay’s article 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 485-503, August.
    4. Zhiqi Wang & Ronald Rousseau, 2021. "COVID-19, the Yule-Simpson paradox and research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3501-3511, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    2. M. Zitt, 2011. "Behind citing-side normalization of citations: some properties of the journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 329-344, October.
    3. Michel Zitt, 2012. "The journal impact factor: angel, devil, or scapegoat? A comment on J.K. Vanclay’s article 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 485-503, August.
    4. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    5. Waltman, Ludo & van Eck, Nees Jan, 2013. "A systematic empirical comparison of different approaches for normalizing citation impact indicators," Journal of Informetrics, Elsevier, vol. 7(4), pages 833-849.
    6. Cristina López-Duarte & Marta M. Vidal-Suárez & Belén González-Díaz, 2019. "Cross-national distance and international business: an analysis of the most influential recent models," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 173-208, October.
    7. Zitt, Michel, 2010. "Citing-side normalization of journal impact: A robust variant of the Audience Factor," Journal of Informetrics, Elsevier, vol. 4(3), pages 392-406.
    8. Theresa Velden & Asif-ul Haque & Carl Lagoze, 2010. "A new approach to analyzing patterns of collaboration in co-authorship networks: mesoscopic analysis and interpretation," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 219-242, October.
    9. Tahamtan, Iman & Bornmann, Lutz, 2018. "Creativity in science and the link to cited references: Is the creative potential of papers reflected in their cited references?," Journal of Informetrics, Elsevier, vol. 12(3), pages 906-930.
    10. Ludo Waltman & Nees Jan Eck, 2013. "Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(3), pages 699-716, September.
    11. Michel Zitt & Suzy Ramanana-Rahary & Elise Bassecoulard, 2003. "Bridging citation and reference distributions: Part I - The referencing-structure function and its application to co-citation and co-item studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 57(1), pages 93-118, May.
    12. Eustache Mêgnigbêto, 2018. "Correlation Between Transmission Power and Some Indicators Used to Measure the Knowledge-Based Economy: Case of Six OECD Countries," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 9(4), pages 1168-1183, December.
    13. José Monteiro-Barata, 2005. "Innovation in the Portuguese Manufacturing Industry: Analysis of a Longitudinal Company Panel," International Advances in Economic Research, Springer;International Atlantic Economic Society, vol. 11(3), pages 301-314, August.
    14. Cristina López-Duarte & Jane F. Maley & Marta M. Vidal-Suárez, 2021. "Main challenges to international student mobility in the European arena," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8957-8980, November.
    15. Seongkyoon Jeong & Jong-Chan Kim & Jae Young Choi, 2015. "Technology convergence: What developmental stage are we in?," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 841-871, September.
    16. Battke, Benedikt & Schmidt, Tobias S. & Stollenwerk, Stephan & Hoffmann, Volker H., 2016. "Internal or external spillovers—Which kind of knowledge is more likely to flow within or across technologies," Research Policy, Elsevier, vol. 45(1), pages 27-41.
    17. Philippe Aghion & Antoine Dechezleprêtre & David Hémous & Ralf Martin & John Van Reenen, 2016. "Carbon Taxes, Path Dependency, and Directed Technical Change: Evidence from the Auto Industry," Journal of Political Economy, University of Chicago Press, vol. 124(1), pages 1-51.
    18. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    19. Yves Gingras & Mahdi Khelfaoui, 2018. "Assessing the effect of the United States’ “citation advantage” on other countries’ scientific impact as measured in the Web of Science (WoS) database," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 517-532, February.
    20. Blind, Knut & Grupp, Hariolf, 1999. "Interdependencies between the science and technology infrastructure and innovation activities in German regions: empirical findings and policy consequences," Research Policy, Elsevier, vol. 28(5), pages 451-468, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:79:y:2009:i:2:d:10.1007_s11192-009-0420-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.