IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0163702.html
   My bibliography  Save this article

Outcome Reporting Bias in Government-Sponsored Policy Evaluations: A Qualitative Content Analysis of 13 Studies

Author

Listed:
  • Arnaud Vaganay

Abstract

The reporting of evaluation outcomes can be a point of contention between evaluators and policy-makers when a given reform fails to fulfil its promises. Whereas evaluators are required to report outcomes in full, policy-makers have a vested interest in framing these outcomes in a positive light–especially when they previously expressed a commitment to the reform. The current evidence base is limited to a survey of policy evaluators, a study on reporting bias in education research and several studies investigating the influence of industry sponsorship on the reporting of clinical trials. The objective of this study was twofold. Firstly, it aimed to assess the risk of outcome reporting bias (ORB or ‘spin’) in pilot evaluation reports, using seven indicators developed by clinicians. Secondly, it sought to examine how the government’s commitment to a given reform may affect the level of ORB found in the corresponding evaluation report. To answer these questions, 13 evaluation reports were content-analysed, all of which found a non-significant effect of the intervention on its stated primary outcome. These reports were systematically selected from a dataset of 233 pilot and experimental evaluations spanning three policy areas and 13 years of government-commissioned research in the UK. The results show that the risk of ORB is real. Indeed, all studies reviewed here resorted to at least one of the presentational strategies associated with a risk of spin. This study also found a small, negative association between the seniority of the reform’s champion and the risk of ORB in the evaluation of that reform. The publication of protocols and the use of reporting guidelines are recommended.

Suggested Citation

  • Arnaud Vaganay, 2016. "Outcome Reporting Bias in Government-Sponsored Policy Evaluations: A Qualitative Content Analysis of 13 Studies," PLOS ONE, Public Library of Science, vol. 11(9), pages 1-21, September.
    • Handle: RePEc:plo:pone00:0163702
    DOI: 10.1371/journal.pone.0163702
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0163702
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0163702&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0163702?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Weaver, R. Kent, 1986. "The Politics of Blame Avoidance," Journal of Public Policy, Cambridge University Press, vol. 6(4), pages 371-398, October.
    2. Kerry Dwan & Douglas G Altman & Juan A Arnaiz & Jill Bloom & An-Wen Chan & Eugenia Cronin & Evelyne Decullier & Philippa J Easterbrook & Erik Von Elm & Carrol Gamble & Davina Ghersi & John P A Ioannid, 2008. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias," PLOS ONE, Public Library of Science, vol. 3(8), pages 1-31, August.
    3. Ferrán Catalá-López & Gabriel Sanfélix-Gimeno & Manuel Ridao & Salvador Peiró, 2013. "When Are Statins Cost-Effective in Cardiovascular Prevention? A Systematic Review of Sponsorship Bias and Conclusions in Economic Evaluations of Statins," PLOS ONE, Public Library of Science, vol. 8(7), pages 1-1, July.
    4. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    5. repec:mpr:mprres:4994 is not listed on IDEAS
    6. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    7. Kerry Dwan & Carrol Gamble & Paula R Williamson & Jamie J Kirkham & the Reporting Bias Group, 2013. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review," PLOS ONE, Public Library of Science, vol. 8(7), pages 1-37, July.
    8. Maira Bes-Rastrollo & Matthias B Schulze & Miguel Ruiz-Canela & Miguel A Martinez-Gonzalez, 2013. "Financial Conflicts of Interest and Reporting Bias Regarding the Association between Sugar-Sweetened Beverages and Weight Gain: A Systematic Review of Systematic Reviews," PLOS Medicine, Public Library of Science, vol. 10(12), pages 1-9, December.
    9. Kristin Rising & Peter Bacchetti & Lisa Bero, 2008. "Reporting Bias in Drug Trials Submitted to the Food and Drug Administration: Review of Publication and Presentation," PLOS Medicine, Public Library of Science, vol. 5(11), pages 1-10, November.
    10. Koehler, Jonathan J., 1993. "The Influence of Prior Beliefs on Scientific Judgments of Evidence Quality," Organizational Behavior and Human Decision Processes, Elsevier, vol. 56(1), pages 28-55, October.
    11. Michael T M Wang & Greg Gamble & Mark J Bolland & Andrew Grey, 2014. "Press Releases Issued by Supplements Industry Organisations and Non-Industry Organisations in Response to Publication of Clinical Research Findings: A Case-Control Study," PLOS ONE, Public Library of Science, vol. 9(7), pages 1-14, July.
    12. Amélie Yavchitz & Isabelle Boutron & Aida Bafeta & Ibrahim Marroun & Pierre Charles & Jean Mantz & Philippe Ravaud, 2012. "Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort Study," PLOS Medicine, Public Library of Science, vol. 9(9), pages 1-11, September.
    13. Susan Purdon & Nina Stratford & Rebecca Taylor & Lucy Natarajan & Stephen Bell & David Wittenburg, "undated". "Impacts of the Job Retention and Rehabilitation Pilot," Mathematica Policy Research Reports c3dc320c351b4bf7a077e8249, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ana Virgolino & Osvaldo Santos & Joana Costa & Mónica Fialho & Ivo Iavicoli & Tiina Santonen & Hanna Tolonen & Evangelia Samoli & Klea Katsouyanni & Georgios Baltatzis & Flavia Ruggieri & Annalisa Abb, 2021. "Challenges to Evidence Synthesis and Identification of Data Gaps in Human Biomonitoring," IJERPH, MDPI, vol. 18(6), pages 1-11, March.
    2. Su Keng Tan & Wai Keung Leung & Alexander Tin Hong Tang & Roger A Zwahlen, 2017. "Effects of mandibular setback with or without maxillary advancement osteotomies on pharyngeal airways: An overview of systematic reviews," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-20, October.
    3. Jovana Kuzmanovic Pficer & Slobodan Dodic & Vojkan Lazic & Goran Trajkovic & Natasa Milic & Biljana Milicic, 2017. "Occlusal stabilization splint for patients with temporomandibular disorders: Meta-analysis of short and long term effects," PLOS ONE, Public Library of Science, vol. 12(2), pages 1-21, February.
    4. Clovis Mariano Faggion Jr & Yun-Chun Wu & Moritz Scheidgen & Yu-Kang Tu, 2015. "Effect of Risk of Bias on the Effect Size of Meta-Analytic Estimates in Randomized Controlled Trials in Periodontology and Implant Dentistry," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-9, September.
    5. Salvador Angosto & Jerónimo García-Fernández & Irena Valantine & Moisés Grimaldi-Puyana, 2020. "The Intention to Use Fitness and Physical Activity Apps: A Systematic Review," Sustainability, MDPI, vol. 12(16), pages 1-25, August.
    6. Nikolaos Pandis & Padhraig S Fleming & Helen Worthington & Kerry Dwan & Georgia Salanti, 2015. "Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-10, September.
    7. Libing Jiang & Shouyin Jiang & Mao Zhang & Zhongjun Zheng & Yuefeng Ma, 2014. "Albumin versus Other Fluids for Fluid Resuscitation in Patients with Sepsis: A Meta-Analysis," PLOS ONE, Public Library of Science, vol. 9(12), pages 1-21, December.
    8. Kellia Chiu & Quinn Grundy & Lisa Bero, 2017. "‘Spin’ in published biomedical literature: A methodological systematic review," PLOS Biology, Public Library of Science, vol. 15(9), pages 1-16, September.
    9. Mahesh Shumsher Rughooputh & Rui Zeng & Ying Yao, 2015. "Protein Diet Restriction Slows Chronic Kidney Disease Progression in Non-Diabetic and in Type 1 Diabetic Patients, but Not in Type 2 Diabetic Patients: A Meta-Analysis of Randomized Controlled Trials ," PLOS ONE, Public Library of Science, vol. 10(12), pages 1-17, December.
    10. Laith Hussain-Alkhateeb & Tatiana Rivera Ramírez & Axel Kroeger & Ernesto Gozzer & Silvia Runge-Ranzinger, 2021. "Early warning systems (EWSs) for chikungunya, dengue, malaria, yellow fever, and Zika outbreaks: What is the evidence? A scoping review," PLOS Neglected Tropical Diseases, Public Library of Science, vol. 15(9), pages 1-25, September.
    11. Shalin Lee Wan Fei & Khatijah L Abdullah, 2015. "Effect of turning vs. supine position under phototherapy on neonates with hyperbilirubinemia: a systematic review," Journal of Clinical Nursing, John Wiley & Sons, vol. 24(5-6), pages 672-682, March.
    12. Savita Bakhshi & Alison E. While, 2013. "Health Professionals’ Alcohol-Related Professional Practices and the Relationship between Their Personal Alcohol Attitudes and Behavior and Professional Practices: A Systematic Review," IJERPH, MDPI, vol. 11(1), pages 1-31, December.
    13. Finkelstein, Eric A. & Bilger, Marcel & Baid, Drishti, 2019. "Effectiveness and cost-effectiveness of incentives as a tool for prevention of non-communicable diseases: A systematic review," Social Science & Medicine, Elsevier, vol. 232(C), pages 340-350.
    14. João Carlos Belloti & Aldo Okamura & Jordana Scheeren & Flávio Faloppa & Vinícius Ynoe de Moraes, 2019. "A systematic review of the quality of distal radius systematic reviews: Methodology and reporting assessment," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-12, January.
    15. Je-Young Lee & Minkyung Baek, 2023. "Effects of Gamification on Students’ English Language Proficiency: A Meta-Analysis on Research in South Korea," Sustainability, MDPI, vol. 15(14), pages 1-19, July.
    16. Kevin Rudolf & Lea A. L. Dejonghe & Ingo Froböse & Florian Lammer & Lisa-Marie Rückel & Jessica Tetz & Andrea Schaller, 2019. "Effectiveness Studies in Health Promotion: A Review of the Methodological Quality of Studies Reporting Significant Effects on Physical Activity in Working Age Adults," IJERPH, MDPI, vol. 16(5), pages 1-17, March.
    17. Taweewat Wiangkham & Joan Duda & Sayeed Haque & Mohammad Madi & Alison Rushton, 2015. "The Effectiveness of Conservative Management for Acute Whiplash Associated Disorder (WAD) II: A Systematic Review and Meta-Analysis of Randomised Controlled Trials," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-22, July.
    18. Eijgermans, D.G.M. & Fang, Y. & Jansen, D.E.M.C. & Bramer, W.M. & Raat, H. & Jansen, W., 2021. "Individual and contextual determinants of children’s and adolescents’ mental health care use: A systematic review," Children and Youth Services Review, Elsevier, vol. 131(C).
    19. Joseph Pozsgai-Alvarez & Iván Pastor Sanz, 2021. "Mapping the (anti-)corruption field: key topics and changing trends, 1968–2020," Journal of Computational Social Science, Springer, vol. 4(2), pages 851-881, November.
    20. Abigail Stevely & Munyaradzi Dimairo & Susan Todd & Steven A Julious & Jonathan Nicholl & Daniel Hind & Cindy L Cooper, 2015. "An Investigation of the Shortcomings of the CONSORT 2010 Statement for the Reporting of Group Sequential Randomised Controlled Trials: A Methodological Systematic Review," PLOS ONE, Public Library of Science, vol. 10(11), pages 1-20, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0163702. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.