Author
Listed:
- Kerry Dwan
- Douglas G Altman
- Mike Clarke
- Carrol Gamble
- Julian P T Higgins
- Jonathan A C Sterne
- Paula R Williamson
- Jamie J Kirkham
Abstract
: In a systematic review of cohort studies, Kerry Dwan and colleagues examine the evidence for selective reporting and discrepancies in analyses between journal publications and other documents for clinical trials. Background: Most publications about selective reporting in clinical trials have focussed on outcomes. However, selective reporting of analyses for a given outcome may also affect the validity of findings. If analyses are selected on the basis of the results, reporting bias may occur. The aims of this study were to review and summarise the evidence from empirical cohort studies that assessed discrepant or selective reporting of analyses in randomised controlled trials (RCTs). Methods and Findings: A systematic review was conducted and included cohort studies that assessed any aspect of the reporting of analyses of RCTs by comparing different trial documents, e.g., protocol compared to trial report, or different sections within a trial publication. The Cochrane Methodology Register, Medline (Ovid), PsycInfo (Ovid), and PubMed were searched on 5 February 2014. Two authors independently selected studies, performed data extraction, and assessed the methodological quality of the eligible studies. Twenty-two studies (containing 3,140 RCTs) published between 2000 and 2013 were included. Twenty-two studies reported on discrepancies between information given in different sources. Discrepancies were found in statistical analyses (eight studies), composite outcomes (one study), the handling of missing data (three studies), unadjusted versus adjusted analyses (three studies), handling of continuous data (three studies), and subgroup analyses (12 studies). Discrepancy rates varied, ranging from 7% (3/42) to 88% (7/8) in statistical analyses, 46% (36/79) to 82% (23/28) in adjusted versus unadjusted analyses, and 61% (11/18) to 100% (25/25) in subgroup analyses. This review is limited in that none of the included studies investigated the evidence for bias resulting from selective reporting of analyses. It was not possible to combine studies to provide overall summary estimates, and so the results of studies are discussed narratively. Conclusions: Discrepancies in analyses between publications and other study documentation were common, but reasons for these discrepancies were not discussed in the trial reports. To ensure transparency, protocols and statistical analysis plans need to be published, and investigators should adhere to these or explain discrepancies. Background: In the past, clinicians relied on their own experience when choosing the best treatment for their patients. Nowadays, they turn to evidence-based medicine—the systematic review and appraisal of trials, studies that investigate the benefits and harms of medical treatments in patients. However, evidence-based medicine can guide clinicians only if all the results from clinical trials are published in an unbiased and timely manner. Unfortunately, the results of trials in which a new drug performs better than existing drugs are more likely to be published than those in which the new drug performs badly or has unwanted side effects (publication bias). Moreover, trial outcomes that support the use of a new treatment are more likely to be published than those that do not support its use (outcome reporting bias). Recent initiatives—such as making registration of clinical trials in a trial registry (for example, ClinicalTrials.gov) a prerequisite for publication in medical journals—aim to prevent these biases, which pose a threat to informed medical decision-making. Why Was This Study Done?: Selective reporting of analyses of outcomes may also affect the validity of clinical trial findings. Sometimes, for example, a trial publication will include a per protocol analysis (which considers only the outcomes of patients who received their assigned treatment) rather than a pre-planned intention-to-treat analysis (which considers the outcomes of all the patients regardless of whether they received their assigned treatment). If the decision to publish the per protocol analysis is based on the results of this analysis being more favorable than those of the intention-to-treat analysis (which more closely resembles “real” life), then “analysis reporting bias” has occurred. In this systematic review, the researchers investigate the selective reporting of analyses and discrepancies in randomized controlled trials (RCTs) by reviewing published studies that assessed selective reporting of analyses in groups (cohorts) of RCTs and discrepancies in analyses of RCTs between different sources (for example, between the protocol in a trial registry and the journal publication) or different sections of a source. A systematic review uses predefined criteria to identify all the research on a given topic. What Did the Researchers Do and Find?: The researchers identified 22 cohort studies (containing 3,140 RCTs) that were eligible for inclusion in their systematic review. All of these studies reported on discrepancies between the information provided by the RCTs in different places, but none investigated the evidence for analysis reporting bias. Several of the cohort studies reported, for example, that there were discrepancies in the statistical analyses included in the different documents associated with the RCTs included in their analysis. Other types of discrepancies reported by the cohort studies included discrepancies in the reporting of composite outcomes (an outcome in which multiple end points are combined) and in the reporting of subgroup analyses (investigations of outcomes in subgroups of patients that should be predefined in the trial protocol to avoid bias). Discrepancy rates varied among the RCTs according to the types of analyses and cohort studies considered. Thus, whereas in one cohort study discrepancies were present in the statistical test used for the analysis of the primary outcome in only 7% of the included studies, they were present in the subgroup analyses of all the included studies. What Do These Findings Mean?: These findings indicate that discrepancies in analyses between publications and other study documents such as protocols in trial registries are common. The reasons for these discrepancies in analyses were not discussed in trial reports but may be the result of reporting bias, errors, or legitimate departures from a pre-specified protocol. For example, a statistical analysis that is not specified in the trial protocol may sometimes appear in a publication because the journal requested its inclusion as a condition of publication. The researchers suggest that it may be impossible for systematic reviewers to distinguish between these possibilities simply by looking at the source documentation. Instead, they suggest, it may be necessary for reviewers to contact the trial authors. However, to make selective reporting of analyses more easily detectable, they suggest that protocols and analysis plans should be published and that investigators should be required to stick to these plans or explain any discrepancies when they publish their trial results. Together with other initiatives, this approach should help improve the quality of evidence-based medicine and, as a result, the treatment of patients. Additional Information: Please access these websites via the online version of this summary at http://dx.doi.org/10.1371/journal.pmed.1001666.
Suggested Citation
Kerry Dwan & Douglas G Altman & Mike Clarke & Carrol Gamble & Julian P T Higgins & Jonathan A C Sterne & Paula R Williamson & Jamie J Kirkham, 2014.
"Evidence for the Selective Reporting of Analyses and Discrepancies in Clinical Trials: A Systematic Review of Cohort Studies of Clinical Trials,"
PLOS Medicine, Public Library of Science, vol. 11(6), pages 1-22, June.
Handle:
RePEc:plo:pmed00:1001666
DOI: 10.1371/journal.pmed.1001666
Download full text from publisher
Citations
Citations are extracted by the
CitEc Project, subscribe to its
RSS feed for this item.
Cited by:
- Nikolaos Pandis & Padhraig S Fleming & Helen Worthington & Kerry Dwan & Georgia Salanti, 2015.
"Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews,"
PLOS ONE, Public Library of Science, vol. 10(9), pages 1-10, September.
- Christoffer Bruun Korfitsen & Marie-Louise Kirkegaard Mikkelsen & Anja Ussing & Karen Christina Walker & Jeanett Friis Rohde & Henning Keinke Andersen & Simon Tarp & Mina Nicole Händel, 2022.
"Usefulness of Cochrane Reviews in Clinical Guideline Development—A Survey of 585 Recommendations,"
IJERPH, MDPI, vol. 19(2), pages 1-10, January.
- Gil Amarilyo & Daniel E Furst & Jennifer M P Woo & Wen Li & Henning Bliddal & Robin Christensen & Simon Tarp, 2016.
"Agreements and Discrepancies between FDA Reports and Journal Papers on Biologic Agents Approved for Rheumatoid Arthritis: A Meta-Research Project,"
PLOS ONE, Public Library of Science, vol. 11(1), pages 1-13, January.
- Julia H. Littell, 2024.
"The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations,"
Evaluation Review, , vol. 48(3), pages 427-460, June.
- Takeshi Seta & Yoshimitsu Takahashi & Yoshinori Noguchi & Satoru Shikata & Tatsuya Sakai & Kyoko Sakai & Yukitaka Yamashita & Takeo Nakayama, 2017.
"Effectiveness of Helicobacter pylori eradication in the prevention of primary gastric cancer in healthy asymptomatic people: A systematic review and meta-analysis comparing risk ratio with risk differ,"
PLOS ONE, Public Library of Science, vol. 12(8), pages 1-18, August.
- Julia H. Littell & Therese D. Pigott & Karianne H. Nilsen & Jennifer Roberts & Travis K. Labrum, 2023.
"Functional Family Therapy for families of youth (age 11–18) with behaviour problems: A systematic review and meta‐analysis,"
Campbell Systematic Reviews, John Wiley & Sons, vol. 19(3), September.
Corrections
All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pmed00:1001666. See general information about how to correct material in RePEc.
If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.
We have no bibliographic references for this item. You can help adding them by using this form .
If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosmedicine (email available below). General contact details of provider: https://journals.plos.org/plosmedicine/ .
Please note that corrections may take a couple of weeks to filter through
the various RePEc services.