IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v92y2012i2d10.1007_s11192-012-0670-4.html
   My bibliography  Save this article

The generalized propensity score methodology for estimating unbiased journal impact factors

Author

Listed:
  • Rüdiger Mutz

    (ETH Zurich)

  • Hans-Dieter Daniel

    (ETH Zurich
    University of Zurich)

Abstract

The journal impact factor (JIF) proposed by Garfield in the year 1955 is one of the most commonly used and prominent citation-based indicators of the performance and significance of a scientific journal. The JIF is simple, reasonable, clearly defined, and comparable over time and, what is more, can be easily calculated from data provided by Thomson Reuters, but at the expense of serious technical and methodological flaws. The paper discusses one of the core problems: The JIF is affected by bias factors (e.g., document type) that have nothing to do with the prestige or quality of a journal. For solving this problem, we suggest using the generalized propensity score methodology based on the Rubin Causal Model. Citation data for papers of all journals in the ISI subject category “Microscopy” (Journal Citation Report) are used to illustrate the proposal.

Suggested Citation

  • Rüdiger Mutz & Hans-Dieter Daniel, 2012. "The generalized propensity score methodology for estimating unbiased journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 377-390, August.
    • Handle: RePEc:spr:scient:v:92:y:2012:i:2:d:10.1007_s11192-012-0670-4
    DOI: 10.1007/s11192-012-0670-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-012-0670-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-012-0670-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Lu, Bo & Greevy, Robert & Xu, Xinyi & Beck, Cole, 2011. "Optimal Nonbipartite Matching and Its Statistical Applications," The American Statistician, American Statistical Association, vol. 65(1), pages 21-30.
    2. Donald B. Rubin, 2004. "Teaching Statistical Inference for Causal Effects in Experiments and Observational Studies," Journal of Educational and Behavioral Statistics, , vol. 29(3), pages 343-367, September.
    3. Wolfgang Glänzel & Henk F. Moed, 2002. "Journal impact measures in bibliometric research," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 171-193, February.
    4. Rosenbaum, Paul R., 2010. "Design Sensitivity and Efficiency in Observational Studies," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 692-702.
    5. Loet Leydesdorff & Lutz Bornmann, 2011. "How fractional counting of citations affects the impact factor: Normalization in terms of differences in citation potentials among fields of science," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 217-229, February.
    6. Mutz, Rüdiger & Daniel, Hans-Dieter, 2012. "Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor," Journal of Informetrics, Elsevier, vol. 6(2), pages 169-176.
    7. Christoph Neuhaus & Werner Marx & Hans‐Dieter Daniel, 2009. "The publication and citation impact profiles of Angewandte Chemie and the Journal of the American Chemical Society based on the sections of Chemical Abstracts: A case study on the limitations of the J," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 176-183, January.
    8. Jochen Kluve & Hilmar Schneider & Arne Uhlendorff & Zhong Zhao, 2007. "Evaluating Continuous Training Programs Using the Generalized Propensity Score," Discussion Papers of DIW Berlin 752, DIW Berlin, German Institute for Economic Research.
    9. H. F. Moed & Th. N. Leeuwen & J. Reedijk, 1999. "Towards appropriate indicators of journal impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 46(3), pages 575-589, November.
    10. Jochen Kluve & Hilmar Schneider & Arne Uhlendorff & Zhong Zhao, 2012. "Evaluating continuous training programmes by using the generalized propensity score," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(2), pages 587-617, April.
    11. Loet Leydesdorff & Jung C. Shin, 2011. "How to evaluate universities in terms of their relative citation impacts: Fractional counting of citations and the normalization of differences among disciplines," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(6), pages 1146-1155, June.
    12. Donald B. Rubin, 2005. "Causal Inference Using Potential Outcomes: Design, Modeling, Decisions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 322-331, March.
    13. Donald B. Rubin, 1977. "Assignment to Treatment Group on the Basis of a Covariate," Journal of Educational and Behavioral Statistics, , vol. 2(1), pages 1-26, March.
    14. Elaine Zanutto & Bo Lu & Robert Hornik, 2005. "Using Propensity Score Subclassification for Multiple Treatment Doses to Evaluate a National Antidrug Media Campaign," Journal of Educational and Behavioral Statistics, , vol. 30(1), pages 59-73, March.
    15. Kosuke Imai & David A. van Dyk, 2004. "Causal Inference With General Treatment Regimes: Generalizing the Propensity Score," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 854-866, January.
    16. Jochen Kluve & Hilmar Schneider & Arne Uhlendorff & Zhong Zhao, 2012. "Evaluating continuous training programmes by using the generalized propensity score," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(2), pages 587-617, April.
    17. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    2. Rüdiger Mutz & Tobias Wolbring & Hans-Dieter Daniel, 2017. "The effect of the “very important paper” (VIP) designation in Angewandte Chemie International Edition on citation impact: A propensity score matching analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2139-2153, September.
    3. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    4. Fernando A. B. Colugnati & Sergio Firpo & Paula F. Drummond Castro & Juan E. Sepulveda & Sergio L. M. Salles-Filho, 2014. "A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 85-107, October.
    5. Xu, Shuqi & Mariani, Manuel Sebastian & Lü, Linyuan & Medo, Matúš, 2020. "Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data," Journal of Informetrics, Elsevier, vol. 14(1).
    6. Adriana Bin & Sergio Salles-Filho & Luiza Maria Capanema & Fernando Antonio Basile Colugnati, 2015. "What difference does it make? Impact of peer-reviewed scholarships on scientific production," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1167-1188, February.
    7. Pilar Valderrama & Manuel Escabias & Evaristo Jiménez-Contreras & Alberto Rodríguez-Archilla & Mariano J. Valderrama, 2018. "Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: application to the field of Dentistry," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1087-1095, May.
    8. Juan Miguel Campanario, 2014. "The effect of citations on the significance of decimal places in the computation of journal impact factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 99(2), pages 289-298, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Swen Kuh & Grace S. Chiu & Anton H. Westveld, 2020. "Latent Causal Socioeconomic Health Index," Papers 2009.12217, arXiv.org, revised Oct 2023.
    3. Noémi Kreif & Richard Grieve & Iván Díaz & David Harrison, 2015. "Evaluation of the Effect of a Continuous Treatment: A Machine Learning Approach with an Application to Treatment for Traumatic Brain Injury," Health Economics, John Wiley & Sons, Ltd., vol. 24(9), pages 1213-1228, September.
    4. Ruth T. Chepchirchir & Ibrahim Macharia & Alice W. Murage & Charles A. O. Midega & Zeyaur R. Khan, 2017. "Impact assessment of push-pull pest management on incomes, productivity and poverty among smallholder households in Eastern Uganda," Food Security: The Science, Sociology and Economics of Food Production and Access to Food, Springer;The International Society for Plant Pathology, vol. 9(6), pages 1359-1372, December.
    5. Kreif, N. & Grieve, R. & Díaz, I. & Harrison, D., 2014. "Health econometric evaluation of the effects of a continuous treatment: a machine learning approach," Health, Econometrics and Data Group (HEDG) Working Papers 14/19, HEDG, c/o Department of Economics, University of York.
    6. Rüdiger Mutz & Tobias Wolbring & Hans-Dieter Daniel, 2017. "The effect of the “very important paper” (VIP) designation in Angewandte Chemie International Edition on citation impact: A propensity score matching analysis," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2139-2153, September.
    7. Hilal Atasoy & Rajiv D. Banker & Paul A. Pavlou, 2016. "On the Longitudinal Effects of IT Use on Firm-Level Employment," Information Systems Research, INFORMS, vol. 27(1), pages 6-26, March.
    8. Ida D'Attoma & Silvia Pacei, 2018. "Evaluating the Effects of Product Innovation on the Performance of European Firms by Using the Generalised Propensity Score," German Economic Review, Verein für Socialpolitik, vol. 19(1), pages 94-112, February.
    9. Steckenleiter, Carina & Lechner, Michael & Pawlowski, Tim & Schüttoff, Ute, 2019. "Do local public expenditures on sports facilities affect sports participation in Germany?," Economics Working Paper Series 1905, University of St. Gallen, School of Economics and Political Science.
    10. A. Giffin & B. J. Reich & S. Yang & A. G. Rappold, 2023. "Generalized propensity score approach to causal inference with spatial interference," Biometrics, The International Biometric Society, vol. 79(3), pages 2220-2231, September.
    11. Mutz, Rüdiger & Daniel, Hans-Dieter, 2012. "Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor," Journal of Informetrics, Elsevier, vol. 6(2), pages 169-176.
    12. Emiliano Magrini & Pierluigi Montalbano & Silvia Nenci, 2013. "Are the EU trade preferences really effective? A Generalized Propensity Score evaluation of the Southern Mediterranean Countries' case in agriculture and fishery," Working Papers 2/13, Sapienza University of Rome, DISS.
    13. Guadalupe Serrano-Domingo & Francisco Requena-Silvente, 2013. "Examining the non-linear relationship between migration and trade," Working Papers 1310, Department of Applied Economics II, Universidad de Valencia.
    14. Tübbicke Stefan, 2022. "Entropy Balancing for Continuous Treatments," Journal of Econometric Methods, De Gruyter, vol. 11(1), pages 71-89, January.
    15. Magrini, Emiliano & Montalbano, Pierluigi & Nenci, Silvia & Salvatici, Luca, 2014. "Agricultural trade distortions during recent international price spikes: what implications for food security?," 2014 International Congress, August 26-29, 2014, Ljubljana, Slovenia 182726, European Association of Agricultural Economists.
    16. Juan M. Villa, 2014. "The length of exposure to antipoverty transfer programmes: what is the relevance for children's human capital formation?," Global Development Institute Working Paper Series 20614, GDI, The University of Manchester.
    17. Chepchirchir, R. & Macharia, I. & Murage, A.W. & Midega, C.A.O. & Khan, Z.R., 2016. "Impact assessment of push-pull technology on incomes, productivity and poverty among smallholder households in Eastern Uganda," 2016 Fifth International Conference, September 23-26, 2016, Addis Ababa, Ethiopia 246316, African Association of Agricultural Economists (AAAE).
    18. Michela Bia & Alessandra Mattei, 2012. "Assessing the effect of the amount of financial aids to Piedmont firms using the generalized propensity score," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 21(4), pages 485-516, November.
    19. Pilar Valderrama & Manuel Escabias & Evaristo Jiménez-Contreras & Alberto Rodríguez-Archilla & Mariano J. Valderrama, 2018. "Proposal of a stochastic model to determine the bibliometric variables influencing the quality of a journal: application to the field of Dentistry," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1087-1095, May.
    20. Carlos A. Flores & Oscar A. Mitnik, 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," Working Papers 2010-10, University of Miami, Department of Economics.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:92:y:2012:i:2:d:10.1007_s11192-012-0670-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.