IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v127y2022i12d10.1007_s11192-022-04408-4.html
   My bibliography  Save this article

A systematic method for identifying references to academic research in grey literature

Author

Listed:
  • Matthew S. Bickley

    (University of Wolverhampton)

  • Kayvan Kousha

    (University of Wolverhampton)

  • Michael Thelwall

    (University of Wolverhampton)

Abstract

Grey literature encompasses documents not published in academic journals or books. Some grey literature has substantial societal importance, such as medical guidelines, government analyses and pressure group reports. Academic research cited in such documents may therefore have had indirect societal impact, such as in policy making, clinical practice or legislation. Identifying citations to academic research from grey literature may therefore help assess its societal impacts. This is difficult, however, due to the variety of document and referencing formats used in grey literature, even from a single organisation. In response, this study introduces and tests a semi-automatic method to match academic journal articles with unstandardised grey literature cited references. For this, the metadata (lead author last name, title, year) of 2.45 million UK Russell Group university outputs was matched against a 100-document sample of UK government grey literature to assess the accuracy of 21 matching heuristics. The optimal method (lead author last name and title in either order, maximum of 200 characters apart) is sufficiently accurate and scalable to make the task of matching research outputs to grey literature references feasible. The method was then applied to 3347 government publications, showing approximately 23% of UK government grey literature in this study contained at least one reference to UK Russell Group university output, and of this grey literature, an average of 3.79 references were present per document. The applied method also shows that economics and environmental science academic research is most cited between 2010 and 2018.

Suggested Citation

  • Matthew S. Bickley & Kayvan Kousha & Michael Thelwall, 2022. "A systematic method for identifying references to academic research in grey literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 6913-6933, December.
    • Handle: RePEc:spr:scient:v:127:y:2022:i:12:d:10.1007_s11192-022-04408-4
    DOI: 10.1007/s11192-022-04408-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-022-04408-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-022-04408-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Werner Marx & Lutz Bornmann, 2016. "Change of perspective: bibliometrics from the point of view of cited references—a literature overview on approaches to the evaluation of cited references in bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1397-1415, November.
    2. Matthew S. Bickley & Kayvan Kousha & Michael Thelwall, 2020. "Can the impact of grey literature be assessed? An investigation of UK government publications cited by articles and books," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1425-1444, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Robin Haunschild & Rüdiger Mutz, 2021. "Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-15, December.
    2. Matthieu Ballandonne & Igor Cersosimo, 2021. "A note on reference publication year spectroscopy with incomplete information," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4927-4939, June.
    3. Kameliya Deyanova & Nataliia Brehmer & Artur Lapidus & Victor Tiberius & Steve Walsh, 2022. "Hatching start-ups for sustainable growth: a bibliometric review on business incubators," Review of Managerial Science, Springer, vol. 16(7), pages 2083-2109, October.
    4. Ksenia Silchenko, 2018. "The other "meta" of meta-analysis: Qualitative and text-based approaches to "analysis of analyses" in marketing," MERCATI & COMPETITIVIT?, FrancoAngeli Editore, vol. 2018(4), pages 27-45.
    5. Anton Gruber & Alexander Tekles & Lutz Bornmann, 2023. "John Mearsheimer’s academic roots: a reference publication year spectroscopy of a political scientist’s oeuvre," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(7), pages 3867-3877, July.
    6. Robin Haunschild & Werner Marx, 2022. "On Health Effects of Resveratrol in Wine," IJERPH, MDPI, vol. 19(5), pages 1-13, March.
    7. Fatemeh Ghaffari & Mark C. Wilson, 2023. "A model for reference list length of scholarly articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(9), pages 5335-5350, September.
    8. Sharma, Anuj & Nunkoo, Robin & Rana, Nripendra P. & Dwivedi, Yogesh K., 2021. "On the intellectual structure and influence of tourism social science research," Annals of Tourism Research, Elsevier, vol. 91(C).
    9. Haunschild, Robin & Bornmann, Lutz, 2023. "Which papers cited which tweets? An exploratory analysis based on Scopus data," Journal of Informetrics, Elsevier, vol. 17(2).
    10. Kayvan Kousha & Mike Thelwall & Matthew Bickley, 2022. "The high scholarly value of grey literature before and during Covid-19," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3489-3504, June.
    11. Lina Xu & Steven Dellaportas & Jin Wang, 2022. "A study of interdisciplinary accounting research: analysing the diversity of cited references," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 62(2), pages 2131-2162, June.
    12. Andreas Thor & Lutz Bornmann & Werner Marx & Rüdiger Mutz, 2018. "Identifying single influential publications in a research field: new analysis opportunities of the CRExplorer," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 591-608, July.
    13. Yonis Gulzar & Nisa Eksili & Pınar Celik Caylak & Mohammad Shuaib Mir, 2023. "Sustainability Consciousness Research Trends: A Bibliometric Analysis," Sustainability, MDPI, vol. 15(24), pages 1-26, December.
    14. António Osório & Lutz Bornmann, 2021. "On the disruptive power of small-teams research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 117-133, January.
    15. Vogel, Rick & Hattke, Fabian & Petersen, Jessica, 2017. "Journal rankings in management and business studies: What rules do we play by?," Research Policy, Elsevier, vol. 46(10), pages 1707-1722.
    16. Silva, Emmanuel Marques & Moreira, Rafael de Lacerda & Bortolon, Patricia Maria, 2023. "Mental Accounting and decision making: a systematic literature review," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 107(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:127:y:2022:i:12:d:10.1007_s11192-022-04408-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.