IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v93y2012i3d10.1007_s11192-012-0714-9.html
   My bibliography  Save this article

What is the appropriate length of the publication period over which to assess research performance?

Author

Listed:
  • Giovanni Abramo

    (National Research Council of Italy
    Università degli Studi di Roma “Tor Vergata”)

  • Ciriaco Andrea D’Angelo

    (Università degli Studi di Roma “Tor Vergata”)

  • Tindaro Cicero

    (Università degli Studi di Roma “Tor Vergata”)

Abstract

National research assessment exercises are conducted in different nations over varying periods. The choice of the publication period to be observed has to address often contrasting needs: it has to ensure the reliability of the results issuing from the evaluation, but also reach the achievement of frequent assessments. In this work we attempt to identify which is the most appropriate or optimal publication period to be observed. For this, we analyze the variation of individual researchers’ productivity rankings with the length of the publication period within the period 2003–2008, by the over 30,000 Italian university scientists in the hard sciences. First we analyze the variation in rankings referring to pairs of contiguous and overlapping publication periods, and show that the variations reduce markedly with periods above 3 years. Then we will show the strong randomness of performance rankings over publication periods under 3 years. We conclude that the choice of a 3 year publication period would seem reliable, particularly for physics, chemistry, biology and medicine.

Suggested Citation

  • Giovanni Abramo & Ciriaco Andrea D’Angelo & Tindaro Cicero, 2012. "What is the appropriate length of the publication period over which to assess research performance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 1005-1017, December.
    • Handle: RePEc:spr:scient:v:93:y:2012:i:3:d:10.1007_s11192-012-0714-9
    DOI: 10.1007/s11192-012-0714-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-012-0714-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-012-0714-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Carlos B. Amat, 2008. "Editorial and publication delay of papers submitted to 14 selected Food Research journals. Influence of online posting," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(3), pages 379-389, March.
    2. Trivedi, Pravin K, 1993. "An Analysis of Publication Lags in Econometrics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 8(1), pages 93-100, Jan.-Marc.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2011. "National-scale research performance assessment at the individual level," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 347-364, February.
    4. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "A sensitivity analysis of researchers’ productivity rankings to the time of citation observation," Journal of Informetrics, Elsevier, vol. 6(2), pages 192-201.
    5. Linda Butler, 2003. "Modifying publication practices in response to funding formulas," Research Evaluation, Oxford University Press, vol. 12(1), pages 39-46, April.
    6. A Bhattacharya & H Newhouse, 2010. "Allocative Efficiency and an Incentive Scheme for Research," Discussion Papers 10/02, Department of Economics, University of York.
    7. Quentin L. Burrell, 2002. "Modelling citation age data: Simple graphical methods from reliability theory," Scientometrics, Springer;Akadémiai Kiadó, vol. 55(2), pages 273-285, August.
    8. Lundberg, Jonas, 2007. "Lifting the crown—citation z-score," Journal of Informetrics, Elsevier, vol. 1(2), pages 145-154.
    9. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "A sensitivity analysis of research institutions’ productivity rankings to the time of citation observation," Journal of Informetrics, Elsevier, vol. 6(2), pages 298-306.
    10. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "Assessing the varying level of impact measurement accuracy as a function of the citation window length," Journal of Informetrics, Elsevier, vol. 5(4), pages 659-667.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sameer Kumar & Jariah Mohd. Jan, 2014. "Research collaboration networks of two OIC nations: comparative study between Turkey and Malaysia in the field of ‘Energy Fuels’, 2009–2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 387-414, January.
    2. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Di Costa, Flavia, 2019. "Diversification versus specialization in scientific research: Which strategy pays off?," Technovation, Elsevier, vol. 82, pages 51-57.
    3. Abramo, Giovanni & D'Angelo, CiriacoAndrea & Di Costa, Flavia, 2024. "The moderating role of personal characteristics of authors in the publications’ quality for quantity trade-off," Journal of Informetrics, Elsevier, vol. 18(1).
    4. Peter James Bentley, 2015. "Cross-country differences in publishing productivity of academics in research universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 865-883, January.
    5. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    6. Giannis Karagiannis & Georgia Paschalidou, 2017. "Assessing research effectiveness: a comparison of alternative nonparametric models," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 68(4), pages 456-468, April.
    7. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2014. "How do you define and measure research productivity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1129-1144, November.
    8. Rojko, Katarina & Lužar, Borut, 2022. "Scientific performance across research disciplines: Trends and differences in the case of Slovenia," Journal of Informetrics, Elsevier, vol. 16(2).
    9. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2018. "The effect of multidisciplinary collaborations on research diversification," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 423-433, July.
    10. Andrea Fronzetti Colladon & Ciriaco Andrea D’Angelo & Peter A. Gloor, 2020. "Predicting the future success of scientific publications through social network and semantic analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 357-377, July.
    11. Jayshree Mamtora & Jacqueline K. Wolstenholme & Gaby Haddow, 2014. "Environmental sciences research in northern Australia, 2000–2011: a bibliometric analysis within the context of a national research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 265-281, January.
    12. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2020. "A novel methodology to assess the scientific standing of nations at field level," Journal of Informetrics, Elsevier, vol. 14(1).
    13. Srđan Bojović & Rada Matić & Zorica Popović & Miroslava Smiljanić & Milena Stefanović & Vera Vidaković, 2014. "An overview of forestry journals in the period 2006–2010 as basis for ascertaining research trends," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1331-1346, February.
    14. Soo Jeung Lee & Christian Schneijderberg & Yangson Kim & Isabel Steinhardt, 2021. "Have Academics’ Citation Patterns Changed in Response to the Rise of World University Rankings? A Test Using First-Citation Speeds," Sustainability, MDPI, vol. 13(17), pages 1-19, August.
    15. Karagiannis, Giannis, 2023. "Aggregating faculty members’ research effectiveness to the department or university level: Exact versus approximate solutions," Socio-Economic Planning Sciences, Elsevier, vol. 90(C).
    16. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2018. "Who benefits from a country’s scientific research?," Journal of Informetrics, Elsevier, vol. 12(1), pages 249-258.
    17. Alan Peter Matthews, 2013. "Physics publication productivity in South African universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 69-86, April.
    18. Thabang Lazarus Bambo & Anastassios Pouris, 2020. "Bibliometric analysis of bioeconomy research in South Africa," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 29-51, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Giovanni Abramo & Ciriaco Andrea D’Angelo & Fulvio Viel, 2013. "The suitability of h and g indexes for measuring the research performance of institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 555-570, December.
    3. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "A sensitivity analysis of researchers’ productivity rankings to the time of citation observation," Journal of Informetrics, Elsevier, vol. 6(2), pages 192-201.
    4. Jian Wang, 2013. "Citation time window choice for research impact evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 851-872, March.
    5. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2011. "The dangers of performance-based research funding in non-competitive higher education systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 641-654, June.
    6. Thelwall, Mike, 2018. "Do females create higher impact research? Scopus citations and Mendeley readers for articles from five countries," Journal of Informetrics, Elsevier, vol. 12(4), pages 1031-1041.
    7. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    8. Alona Zharova & Wolfgang K. Härdle & Stefan Lessmann, 2017. "Is Scientific Performance a Function of Funds?," SFB 649 Discussion Papers SFB649DP2017-028, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    9. Giovanni Abramo & Corrado Costa & Ciriaco Andrea D’Angelo, 2015. "A multivariate stochastic model to assess research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1755-1772, February.
    10. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.
    11. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2014. "How do you define and measure research productivity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1129-1144, November.
    12. Vincent Larivière & Rodrigo Costas, 2016. "How Many Is Too Many? On the Relationship between Research Productivity and Impact," PLOS ONE, Public Library of Science, vol. 11(9), pages 1-10, September.
    13. Marcel Clermont & Alexander Dirksen & Barbara Scheidt & Dirk Tunger, 2017. "Citation metrics as an additional indicator for evaluating research performance? An analysis of their correlations and validity," Business Research, Springer;German Academic Association for Business Research, vol. 10(2), pages 249-279, October.
    14. Wang, Xing & Zhang, Zhihui, 2020. "Improving the reliability of short-term citation impact indicators by taking into account the correlation between short- and long-term citation impact," Journal of Informetrics, Elsevier, vol. 14(2).
    15. Zhichao Fang & Rodrigo Costas, 2020. "Studying the accumulation velocity of altmetric data tracked by Altmetric.com," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1077-1101, May.
    16. Björk, Bo-Christer & Solomon, David, 2013. "The publishing delay in scholarly peer-reviewed journals," Journal of Informetrics, Elsevier, vol. 7(4), pages 914-923.
    17. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "Ranking research institutions by the number of highly-cited articles per scientist," Journal of Informetrics, Elsevier, vol. 9(4), pages 915-923.
    18. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2014. "Relatives in the same university faculty: nepotism or merit?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 737-749, October.
    19. Andrea Fronzetti Colladon & Ciriaco Andrea D’Angelo & Peter A. Gloor, 2020. "Predicting the future success of scientific publications through social network and semantic analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 357-377, July.
    20. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "A sensitivity analysis of research institutions’ productivity rankings to the time of citation observation," Journal of Informetrics, Elsevier, vol. 6(2), pages 298-306.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:93:y:2012:i:3:d:10.1007_s11192-012-0714-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.