IDEAS home Printed from https://ideas.repec.org/a/oup/scippl/v49y2022i2p313-323..html
   My bibliography  Save this article

Assessing broader impacts of funded research: the US National Science Foundation v. Lamar Smith
[What is Societal Impact of Research and How Can it Be Assessed? A Literature Survey]

Author

Listed:
  • Caitlin Drummond Otten
  • Baruch Fischhoff

Abstract

The US National Science Foundation (NSF) must demonstrate the value of the research that it funds. To that end, NSF has developed two criteria for its funding decisions: intellectual merit (IM), the potential to advance knowledge, and broader impacts (BI), the potential to benefit society. Critics have argued that these criteria are insufficient, including former Rep. Lamar Smith (R-TX), who offered his own criteria delineating specific desired impacts, such as economic competitiveness. We empirically assess this criticism by comparing public ratings of NSF-funded projects, based on their Project Outcomes Report (POR), using NSF’s and Smith’s criteria. Participants rated NSF-funded research as satisfying both NSF’s and Smith’s criteria, which were moderately correlated. Adding explicit references to societal BI improved ratings slightly. Noting NSF support did not. Our results suggest that having PORs explicitly address additional criteria could increase perceived BI without compromising IM.

Suggested Citation

  • Caitlin Drummond Otten & Baruch Fischhoff, 2022. "Assessing broader impacts of funded research: the US National Science Foundation v. Lamar Smith [What is Societal Impact of Research and How Can it Be Assessed? A Literature Survey]," Science and Public Policy, Oxford University Press, vol. 49(2), pages 313-323.
    • Handle: RePEc:oup:scippl:v:49:y:2022:i:2:p:313-323.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/scipol/scab082
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. J Britt Holbrook & Robert Frodeman, 2011. "Peer review and the ex ante assessment of societal impacts," Research Evaluation, Oxford University Press, vol. 20(3), pages 239-246, September.
    2. Daniel Sarewitz, 2011. "The dubious benefits of broader impact," Nature, Nature, vol. 475(7355), pages 141-141, July.
    3. Salter, Ammon J. & Martin, Ben R., 2001. "The economic benefits of publicly funded basic research: a critical review," Research Policy, Elsevier, vol. 30(3), pages 509-532, March.
    4. Erik Ernø-Kjølhede & Finn Hansson, 2011. "Measuring research performance during a changing relationship between science and society," Research Evaluation, Oxford University Press, vol. 20(2), pages 131-143, June.
    5. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    6. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    7. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    8. Omar Kassab, 2019. "Does public outreach impede research performance? Exploring the ‘researcher’s dilemma’ in a sustainability research center," Science and Public Policy, Oxford University Press, vol. 46(5), pages 710-720.
    9. Bozeman, Barry & Youtie, Jan, 2017. "Socio-economic impacts and public value of government-funded research: Lessons from four US National Science Foundation initiatives," Research Policy, Elsevier, vol. 46(8), pages 1387-1398.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Werner Marx, 2014. "How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 211-219, January.
    2. Irwin Feller, 2022. "Assessing the societal impact of publicly funded research," The Journal of Technology Transfer, Springer, vol. 47(3), pages 632-650, June.
    3. Hashem Atapour & Robabeh Maddahi & Rasoul Zavaraqi, 2024. "Policy citations of scientometric articles: an altmetric study," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4423-4436, July.
    4. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    5. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    6. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    7. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    8. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    9. Matt, M. & Gaunand, A. & Joly, P-B. & Colinet, L., 2017. "Opening the black box of impact – Ideal-type impact pathways in a public agricultural research organization," Research Policy, Elsevier, vol. 46(1), pages 207-218.
    10. Pierre Benoit Joly & Laurence Colinet & Ariane Gaunand & Stephane Lemarié & Mireille Matt, 2016. "Agricultural research impact assessment: issues, methods and challenges," Working Papers hal-01431457, HAL.
    11. Bozeman, Barry & Youtie, Jan, 2017. "Socio-economic impacts and public value of government-funded research: Lessons from four US National Science Foundation initiatives," Research Policy, Elsevier, vol. 46(8), pages 1387-1398.
    12. Nathalie Taverdet-Popiolek, 2022. "Economic Footprint of a Large French Research and Technology Organisation in Europe: Deciphering a Simplified Model and Appraising the Results," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 13(1), pages 44-69, March.
    13. Castelnovo, Paolo & Florio, Massimo & Forte, Stefano & Rossi, Lucio & Sirtori, Emanuela, 2018. "The economic impact of technological procurement for large-scale research infrastructures: Evidence from the Large Hadron Collider at CERN," Research Policy, Elsevier, vol. 47(9), pages 1853-1867.
    14. Alba Viana Lora & Marta Gemma Nel-lo Andreu, 2020. "Alternative Metrics for Assessing the Social Impact of Tourism Research," Sustainability, MDPI, vol. 12(10), pages 1-12, May.
    15. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    16. Arnott, James C., 2021. "Pens and purse strings: Exploring the opportunities and limits to funding actionable sustainability science," Research Policy, Elsevier, vol. 50(10).
    17. Oscar LLOPIS & Joaquin AZAGRA-CARO, 2015. "Who do you care about? Scientists’ personality traits and perceived beneficiary impact," Cahiers du GREThA (2007-2019) 2015-29, Groupe de Recherche en Economie Théorique et Appliquée (GREThA).
    18. Hans Jonker & Florian Vanlee & Walter Ysebaert, 2022. "Societal impact of university research in the written press: media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7289-7306, December.
    19. Rossi, Federica & De Silva, Muthu & Pavone, Pasquale & Rosli, Ainurul & Yip, Nick K.T., 2024. "Proximity and impact of university-industry collaborations. A topic detection analysis of impact reports," Technological Forecasting and Social Change, Elsevier, vol. 205(C).
    20. Tommaso Agasisti & Cristian Barra & Roberto Zotti, 2019. "Research, knowledge transfer, and innovation: The effect of Italian universities’ efficiency on local economic development 2006−2012," Journal of Regional Science, Wiley Blackwell, vol. 59(5), pages 819-849, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:scippl:v:49:y:2022:i:2:p:313-323.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/spp .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.