IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v27y2018i1p16-27..html
   My bibliography  Save this article

Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good enough’

Author

Listed:
  • Barbara L Riley
  • Alison Kernoghan
  • Lisa Stockton
  • Steve Montague
  • Jennifer Yessis
  • Cameron D Willis

Abstract

Assessing societal impacts of research is more difficult than assessing advances in knowledge. Methods to evaluate research impact on policy processes and outcomes are especially underdeveloped, and are needed to optimize the influence of research on policy for addressing complex issues such as chronic diseases. Contribution analysis (CA), a theory-based approach to evaluation, holds promise under these conditions of complexity. Yet applications of CA for this purpose are limited, and methods are needed to strengthen contribution claims and ensure CA is practical to implement. This article reports the experience of a public health research center in Canada that applied CA to evaluate the impacts of its research on policy changes. The main goal was to experiment with methods that were relevant to CA objectives, sufficiently rigorous for making credible claims, and feasible. Methods were ‘good enough’ if they achieved all three attributes. Three cases on government policy in tobacco control were examined: creation of smoke-free multiunit dwellings, creation of smoke-free outdoor spaces, and regulation of flavored tobacco products. Getting to ‘good enough’ required careful selection of nested theories of change; strategic use of social science theories, as well as quantitative and qualitative data from diverse sources; and complementary methods to assemble and analyze evidence for testing the nested theories of change. Some methods reinforced existing good practice standards for CA, and others were adaptations or extensions of them. Our experience may inform efforts to influence policy with research, evaluate research impacts on policy using CA, and apply CA more broadly.

Suggested Citation

  • Barbara L Riley & Alison Kernoghan & Lisa Stockton & Steve Montague & Jennifer Yessis & Cameron D Willis, 2018. "Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good enough’," Research Evaluation, Oxford University Press, vol. 27(1), pages 16-27.
  • Handle: RePEc:oup:rseval:v:27:y:2018:i:1:p:16-27.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvx037
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Annette Boaz & Siobhan Fitzpatrick & Ben Shaw, 2009. "Assessing the impact of research on policy: A literature review," Science and Public Policy, Oxford University Press, vol. 36(4), pages 255-270, May.
    2. Steve Montague & Rodolfo Valentim, 2010. "Evaluation of RT&D: from ‘prescriptions for justifying’ to ‘user-oriented guidance for learning’," Research Evaluation, Oxford University Press, vol. 19(4), pages 251-261, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Genowefa Blundo-Canto & Bernard Triomphe & Guy Faure & Danielle Barret & Aurelle de Romemont & Etienne Hainzelin, 2019. "Building a culture of impact in an international agricultural research organization: Process and reflective learning," Research Evaluation, Oxford University Press, vol. 28(2), pages 136-144.
    2. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    3. D. J. H. te Lintelo & K. Pittore, 2021. "Evaluating Parliamentary Advocacy for Nutrition in Tanzania," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 33(3), pages 735-759, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Isabel Vogel & Chris Barnett, 2023. "Laying the Foundations for Impact: Lessons from the GCRF Evaluation," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 35(2), pages 281-297, April.
    2. Molly Morgan Jones & Catriona Manville & Joanna Chataway, 2022. "Learning from the UK’s research impact assessment exercise: a case study of a retrospective impact assessment exercise and questions for the future," The Journal of Technology Transfer, Springer, vol. 47(3), pages 722-746, June.
    3. Borst, Robert A.J. & Kok, Maarten Olivier & O’Shea, Alison J. & Pokhrel, Subhash & Jones, Teresa H. & Boaz, Annette, 2019. "Envisioning and shaping translation of knowledge into action: A comparative case-study of stakeholder engagement in the development of a European tobacco control tool," Health Policy, Elsevier, vol. 123(10), pages 917-923.
    4. Heyeres, Marion & Tsey, Komla & Yang, Yinghong & Yan, Li & Jiang, Hua, 2019. "The characteristics and reporting quality of research impact case studies: A systematic review," Evaluation and Program Planning, Elsevier, vol. 73(C), pages 10-23.
    5. Jesper Dahl Kelstrup & Jonas Videbæk Jørgensen, 2024. "Explaining differences in research utilization in evidence-based government ministries," Policy Sciences, Springer;Society of Policy Sciences, vol. 57(2), pages 257-280, June.
    6. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    7. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    8. A. Gaunand & L. Colinet & P.-B. Joly & M. Matt, 2022. "Counting what really counts? Assessing the political impact of science," The Journal of Technology Transfer, Springer, vol. 47(3), pages 699-721, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:27:y:2018:i:1:p:16-27.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.