IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v106y2016i3d10.1007_s11192-015-1828-7.html
   My bibliography  Save this article

Automated Research Impact Assessment: a new bibliometrics approach

Author

Listed:
  • Christina H. Drew

    (National Institute of Environmental Health Sciences)

  • Kristianna G. Pettibone

    (National Institute of Environmental Health Sciences)

  • Fallis Owen Finch

    (Open Intelligence, Inc.)

  • Douglas Giles

    (Open Intelligence, Inc.)

  • Paul Jordan

    (National Institutes of Health)

Abstract

As federal programs are held more accountable for their research investments, The National Institute of Environmental Health Sciences (NIEHS) has developed a new method to quantify the impact of our funded research on the scientific and broader communities. In this article we review traditional bibliometric analyses, address challenges associated with them, and describe a new bibliometric analysis method, the Automated Research Impact Assessment (ARIA). ARIA taps into a resource that has only rarely been used for bibliometric analyses: references cited in “important” research artifacts, such as policies, regulations, clinical guidelines, and expert panel reports. The approach includes new statistics that science managers can use to benchmark contributions to research by funding source. This new method provides the ability to conduct automated impact analyses of federal research that can be incorporated in program evaluations. We apply this method to several case studies to examine the impact of NIEHS funded research.

Suggested Citation

  • Christina H. Drew & Kristianna G. Pettibone & Fallis Owen Finch & Douglas Giles & Paul Jordan, 2016. "Automated Research Impact Assessment: a new bibliometrics approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(3), pages 987-1005, March.
    • Handle: RePEc:spr:scient:v:106:y:2016:i:3:d:10.1007_s11192-015-1828-7
    DOI: 10.1007/s11192-015-1828-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-015-1828-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-015-1828-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. T. J. Phelan, 1999. "A compendium of issues for citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 45(1), pages 117-136, May.
    2. Christina Viola Srivastava & Nathaniel Deshmukh Towery & Brian Zuckerman, 2007. "Challenges and opportunities for research portfolio analysis, management, and evaluation," Research Evaluation, Oxford University Press, vol. 16(3), pages 152-156, September.
    3. Isola Ajiferuke & Dietmar Wolfram, 2010. "Citer analysis as a measure of research impact: library and information science as a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(3), pages 623-638, June.
    4. Laurel L. Haak & Will Ferriss & Kevin Wright & Michael E. Pollard & Kirk Barden & Matt A. Probus & Michael Tartakovsky & Charles J. Hackett, 2012. "The electronic Scientific Portfolio Assistant: Integrating scientific knowledge databases to support program impact assessment," Science and Public Policy, Oxford University Press, vol. 39(4), pages 464-475, May.
    5. Julia Lane, 2010. "Let's make science metrics more scientific," Nature, Nature, vol. 464(7288), pages 488-489, March.
    6. Carlyn E Orians & Joanne Abed & Christina H Drew & Shyanika Wijesinha Rose & Jennifer H Cohen & Jerry Phelps, 2009. "Scientific and public health impacts of the NIEHS Extramural Asthma Research Program: insights from primary data," Research Evaluation, Oxford University Press, vol. 18(5), pages 375-385, December.
    7. Grant Lewison & Isla Rippon & Steven Wooding, 2005. "Tracking knowledge diffusion through citations," Research Evaluation, Oxford University Press, vol. 14(1), pages 5-14, April.
    8. Diana Hicks & Hiroyuki Tomizawa & Yoshiko Saitoh & Shinichi Kobayashi, 2004. "Bibliometric techniques in the evaluation of federally funded research in the United States," Research Evaluation, Oxford University Press, vol. 13(2), pages 76-86, August.
    9. Teresa H. Jones & Claire Donovan & Steve Hanney, 2012. "Tracing the wider impacts of biomedical research: a literature search to develop a novel citation categorisation technique," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(1), pages 125-134, October.
    10. Gerard Pasterkamp & Joris I. Rotmans & Dominique V. P. Kleijn & Cornelius Borst, 2007. "Citation frequency: A biased measure of research impact significantly influenced by the geographical origin of research articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 70(1), pages 153-165, January.
    11. Hughes, Dana & Docto, Lindsay & Peters, Jessica & Lamb, Anne Kelsey & Brindis, Claire, 2013. "Swimming upstream: The challenges and rewards of evaluating efforts to address inequities and reduce health disparities," Evaluation and Program Planning, Elsevier, vol. 38(C), pages 1-12.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mike Thelwall & Kayvan Kousha, 2016. "Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1341-1351, November.
    2. Aurora A. C. Teixeira & Pedro Cosme Vieira & Ana Patrícia Abreu, 2017. "Sleeping Beauties and their princes in innovation studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 541-580, February.
    3. Heather Keathley-Herring & Eileen Van Aken & Fernando Gonzalez-Aleu & Fernando Deschamps & Geert Letens & Pablo Cardenas Orlandini, 2016. "Assessing the maturity of a research area: bibliometric review and proposed framework," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 927-951, November.
    4. Cappelletti-Montano, Beniamino & Columbu, Silvia & Montaldo, Stefano & Musio, Monica, 2022. "Interpreting the outcomes of research assessments: A geometrical approach," Journal of Informetrics, Elsevier, vol. 16(1).
    5. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    6. Balázs Győrffy & Andrea Magda Nagy & Péter Herman & Ádám Török, 2018. "Factors influencing the scientific performance of Momentum grant holders: an evaluation of the first 117 research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 409-426, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ashkan Ebadi & Andrea Schiffauerova, 2016. "iSEER: an intelligent automatic computer system for scientific evaluation of researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 477-498, May.
    2. Alexander N. Larcombe & Sasha C. Voss, 2011. "Self-citation: comparison between Radiology, European Radiology and Radiology for 1997–1998," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(2), pages 347-356, May.
    3. Thomas Gurney & Edwin Horlings & Peter van den Besselaar, 2012. "Author disambiguation using multi-aspect similarity indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(2), pages 435-449, May.
    4. Franceschini, Fiorenzo & Maisano, Domenico, 2011. "Structured evaluation of the scientific output of academic research groups by recent h-based indicators," Journal of Informetrics, Elsevier, vol. 5(1), pages 64-74.
    5. Lina Xu & Steven Dellaportas & Zhiqiang Yang & Jin Wang, 2023. "More on the relationship between interdisciplinary accounting research and citation impact," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 63(4), pages 4779-4803, December.
    6. Eduardo A. Haddad & Jesus P. Mena-Chalco, Otávio J.G. Sidone, 2016. "Produção Científica e Redes de Colaboração dos Docentes Vinculados aos Programas de Pós-graduação em Economia no Brasil," Working Papers, Department of Economics 2016_10, University of São Paulo (FEA-USP).
    7. Wen-Yau Cathy Lin & Mu-Hsuan Huang, 2012. "The relationship between co-authorship, currency of references and author self-citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(2), pages 343-360, February.
    8. Guetz, Bernhard & Bidmon, Sonja, 2023. "The Credibility of Physician Rating Websites: A Systematic Literature Review," Health Policy, Elsevier, vol. 132(C).
    9. J Sylvan Katz, 2016. "What Is a Complex Innovation System?," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-24, June.
    10. Stefano Bianchini & Patrick Llerena, 2016. "Science policy as a prerequisite of industrial policy," Economia e Politica Industriale: Journal of Industrial and Business Economics, Springer;Associazione Amici di Economia e Politica Industriale, vol. 43(3), pages 273-280, September.
    11. Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2014. "The citer-success-index: a citer-based indicator to select a subset of elite papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 963-983, November.
    12. Haeussler, Carolin & Sauermann, Henry, 2013. "Credit where credit is due? The impact of project contributions and social factors on authorship and inventorship," Research Policy, Elsevier, vol. 42(3), pages 688-703.
    13. Nelson, Andrew J., 2012. "Putting university research in context: Assessing alternative measures of production and diffusion at Stanford," Research Policy, Elsevier, vol. 41(4), pages 678-691.
    14. Popp, David, 2012. "The role of technological change in green growth," Policy Research Working Paper Series 6239, The World Bank.
    15. Cristina Gomes Souza & Marta Lúcia Azevedo Ferreira, 2013. "Researchers profile, co-authorship pattern and knowledge organization in information science in Brazil," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(2), pages 673-687, May.
    16. Mauro Vitor Mendlowicz & Evandro Silva Freire Coutinho & Jerson Laks & Leonardo Franklin Fontenelle & Alexandre Martins Valença & William Berger & Ivan Figueira & Gláucia Azambuja Aguiar, 2011. "Is there a ‘gender gap’ in authorship of the main Brazilian psychiatric journals at the beginning of the 21st century?," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(1), pages 27-37, January.
    17. Zaggl, Michael A., 2017. "Manipulation of explicit reputation in innovation and knowledge exchange communities: The example of referencing in science," Research Policy, Elsevier, vol. 46(5), pages 970-983.
    18. Chiang Kao & Hsiou-Wei Lin & San-Lin Chung & Wei-Chi Tsai & Jyh-Shen Chiou & Yen-Liang Chen & Liang-Hsuan Chen & Shih-Chieh Fang & Hwei-Lan Pao, 2008. "Ranking Taiwanese management journals: A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 76(1), pages 95-115, July.
    19. Meyer, Martin, 2006. "Are patenting scientists the better scholars?: An exploratory comparison of inventor-authors with their non-inventing peers in nano-science and technology," Research Policy, Elsevier, vol. 35(10), pages 1646-1662, December.
    20. van den Besselaar, Peter & Heyman, Ulf & Sandström, Ulf, 2017. "Perverse effects of output-based research funding? Butler’s Australian case revisited," Journal of Informetrics, Elsevier, vol. 11(3), pages 905-918.

    More about this item

    Keywords

    Bibliometrics; Automated impact analysis; Research evaluation; Science of science management;
    All these keywords.

    JEL classification:

    • I2 - Health, Education, and Welfare - - Education

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:106:y:2016:i:3:d:10.1007_s11192-015-1828-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.