IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/knjea.html
   My bibliography  Save this paper

Replication value as a function of citation impact and sample size

Author

Listed:
  • Isager, Peder Mortvedt

    (Eindhoven University of Technology)

  • van 't Veer, Anna Elisabeth

    (Leiden University)

  • Lakens, Daniel

    (Eindhoven University of Technology)

Abstract

Researchers seeking to replicate original research often need to decide which of several relevant candidates to select for replication. Several strategies for study selection have been proposed, utilizing a variety of observed indicators as criteria for selection. However, few strategies clearly specify the goal of study selection and how that goal is related to the indicators that are utilized. We have previously formalized a decision model of replication study selection in which the goal of study selection is to maximize the expected utility gain of the replication e?ort. We further define the concept of replication value as a proxy for expected utility gain (Isager et al., 2020). In this article, we propose a quantitative operationalization of replication value. We first discuss how value and uncertainty - the two concepts used to determine replication value – could be estimated via information about citation count and sample size. Second, we propose an equation for combining these indicators into an overall estimate of replication value, which we denote RVCn. Third, we suggest how RVCn could be implemented as part of a broader study selection procedure. Finally, we provide preliminary data suggesting that studies that were in fact selected for replication tend to have relatively high RVCn estimates. The goal of this article is to explain how RVCn is intended to work and, in doing so, demonstrate the many assumptions that should be explicit in any replication study selection strategy.

Suggested Citation

  • Isager, Peder Mortvedt & van 't Veer, Anna Elisabeth & Lakens, Daniel, 2021. "Replication value as a function of citation impact and sample size," MetaArXiv knjea, Center for Open Science.
    • Handle: RePEc:osf:metaar:knjea
    DOI: 10.31219/osf.io/knjea
    as

    Download full text from publisher

    File URL: https://osf.io/download/6124a6c087c8f1030fb1356d/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/knjea?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Lisa Klautzer & Stephen Hanney & Edward Nason & Jennifer Rubin & Jonathan Grant & Steven Wooding, 2011. "Assessing policy and practice impacts of social science research: the application of the Payback Framework to assess the Future of Work programme," Research Evaluation, Oxford University Press, vol. 20(3), pages 201-209, September.
    2. Adam Altmejd & Anna Dreber & Eskil Forsell & Juergen Huber & Taisuke Imai & Magnus Johannesson & Michael Kirchler & Gideon Nave & Colin Camerer, 2019. "Predicting the replicability of social science lab experiments," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-18, December.
    3. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    4. Waltman, Ludo & van Eck, Nees Jan, 2013. "A systematic empirical comparison of different approaches for normalizing citation impact indicators," Journal of Informetrics, Elsevier, vol. 7(4), pages 833-849.
    5. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    6. Martín-Martín, Alberto & Orduna-Malea, Enrique & Thelwall, Mike & Delgado López-Cózar, Emilio, 2018. "Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories," Journal of Informetrics, Elsevier, vol. 12(4), pages 1160-1177.
    7. John P. A. Ioannidis & Kevin W. Boyack & Henry Small & Aaron A. Sorensen & Richard Klavans, 2014. "Bibliometrics: Is your most cited work your best?," Nature, Nature, vol. 514(7524), pages 561-562, October.
    8. Radicchi, Filippo & Weissman, Alexander & Bollen, Johan, 2017. "Quantifying perceived impact of scientific publications," Journal of Informetrics, Elsevier, vol. 11(3), pages 704-712.
    9. Parolo, Pietro Della Briotta & Pan, Raj Kumar & Ghosh, Rumi & Huberman, Bernardo A. & Kaski, Kimmo & Fortunato, Santo, 2015. "Attention decay in science," Journal of Informetrics, Elsevier, vol. 9(4), pages 734-745.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    2. Ana Teresa Santos & Sandro Mendonça, 2022. "Do papers (really) match journals’ “aims and scope”? A computational assessment of innovation studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7449-7470, December.
    3. A. Velez-Estevez & P. García-Sánchez & J. A. Moral-Munoz & M. J. Cobo, 2022. "Why do papers from international collaborations get more citations? A bibliometric analysis of Library and Information Science papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7517-7555, December.
    4. Sánchez-Pérez, Manuel & Marín-Carrillo, María Belén & Segovia-López, Cristina & Terán-Yépez, Eduardo, 2025. "Bibliometric articles in business and management: Factors affecting production and scholarly impact," Journal of Business Research, Elsevier, vol. 186(C).
    5. Francisco Grimaldo & Mario Paolucci & Jordi Sabater-Mir, 2018. "Reputation or peer review? The role of outliers," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1421-1438, September.
    6. Wang, Xing & Zhang, Zhihui, 2020. "Improving the reliability of short-term citation impact indicators by taking into account the correlation between short- and long-term citation impact," Journal of Informetrics, Elsevier, vol. 14(2).
    7. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "On the interplay between normalisation, bias, and performance of paper impact metrics," Journal of Informetrics, Elsevier, vol. 13(1), pages 270-290.
    8. Soo Jeung Lee & Christian Schneijderberg & Yangson Kim & Isabel Steinhardt, 2021. "Have Academics’ Citation Patterns Changed in Response to the Rise of World University Rankings? A Test Using First-Citation Speeds," Sustainability, MDPI, vol. 13(17), pages 1-19, August.
    9. Lutz Bornmann & Klaus Wohlrabe, 2024. "Recent Temporal Dynamics in Economics: Empirical Analyses of Annual Publications in Economic Fields," CESifo Working Paper Series 10881, CESifo.
    10. Karin Martín-Bujack & Isabel Figuerola-Ferretti & Teresa Corzo & Ioannis Paraskevopoulos, 2022. "Building Knowledge in the Oil Market," SAGE Open, , vol. 12(1), pages 21582440211, January.
    11. Petersen, Alexander M. & Pan, Raj K. & Pammolli, Fabio & Fortunato, Santo, 2019. "Methods to account for citation inflation in research evaluation," Research Policy, Elsevier, vol. 48(7), pages 1855-1865.
    12. Katchanov, Yurij L. & Markova, Yulia V. & Shmatko, Natalia A., 2023. "Uncited papers in the structure of scientific communication," Journal of Informetrics, Elsevier, vol. 17(2).
    13. de Carvalho, Gustavo Dambiski Gomes & Sokulski, Carla Cristiane & da Silva, Wesley Vieira & de Carvalho, Hélio Gomes & de Moura, Rafael Vignoli & de Francisco, Antonio Carlos & da Veiga, Claudimar Per, 2020. "Bibliometrics and systematic reviews: A comparison between the Proknow-C and the Methodi Ordinatio," Journal of Informetrics, Elsevier, vol. 14(3).
    14. Prodhan Mahbub Ibna Seraj & Blanka Klimova & Rubina Khan, 2024. "Visualizing Research Trends in English Language Teaching (ELT) From 2013 to 2022: A Bibliometric Analysis," SAGE Open, , vol. 14(2), pages 21582440241, May.
    15. Jingqi Gao & Xiang Wu & Xiaowei Luo & Shukai Guan, 2021. "Scientometric Analysis of Safety Sign Research: 1990–2019," IJERPH, MDPI, vol. 18(1), pages 1-15, January.
    16. Alberto Saracco, 2022. "Dr. Strangelove: Or How I Learned to Stop Worrying and Love the Citations," The Mathematical Intelligencer, Springer, vol. 44(4), pages 326-330, December.
    17. Fuentes, Agustín & Espinoza, Ulises J. & Cobbs, Virginia, 2024. "Follow the citations: Tracing pathways of “race as biology” assumptions in medical algorithms in eGFR and spirometry," Social Science & Medicine, Elsevier, vol. 346(C).
    18. Zhichao Wang & Valentin Zelenyuk, 2021. "Performance Analysis of Hospitals in Australia and its Peers: A Systematic Review," CEPA Working Papers Series WP012021, School of Economics, University of Queensland, Australia.
    19. Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    20. Pantea Kamrani & Isabelle Dorsch & Wolfgang G. Stock, 2021. "Do researchers know what the h-index is? And how do they estimate its importance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5489-5508, July.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:knjea. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.