IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0235485.html
   My bibliography  Save this article

Methods used to assess outcome consistency in clinical studies: A literature-based evaluation

Author

Listed:
  • Ewelina Rogozińska
  • Elizabeth Gargon
  • Rocío Olmedo-Requena
  • Amani Asour
  • Natalie A M Cooper
  • Claire L Vale
  • Janneke van’t Hooft

Abstract

Evaluation studies of outcomes used in clinical research and their consistency are appearing more frequently in the literature, as a key part of the core outcome set (COS) development. Current guidance suggests such evaluation studies should use systematic review methodology as their default. We aimed to examine the methods used. We searched the Core Outcome Measures in Effectiveness Trials (COMET) database (up to May 2019) supplementing it with additional resources. We included evaluation studies of outcome consistency in clinical studies across health subjects and used a subset of A MeaSurement Tool to Assess systematic Reviews (AMSTAR) 2 (items 1–9) to assess their methods. Of 93 included evaluation studies of outcome consistency (90 full reports, three summaries), 91% (85/93) reported performing literature searches in at least one bibliographic database, and 79% (73/93) was labelled as a “systematic review”. The evaluations varied in terms of satisfying AMSTAR 2 criteria, such that 81/93 (87%) had implemented PICO in the research question, whereas only 5/93 (6%) had included the exclusions list. None of the evaluation studies explained how inconsistency of outcomes was detected, however, 80/90 (88%) concluded inconsistency in individual outcomes (66%, 55/90) or outcome domains (20%, 18/90). Methods used in evaluation studies of outcome consistency in clinical studies differed considerably. Despite frequent being labelled as a “systematic review”, adoption of systematic review methodology is selective. While the impact on COS development is unknown, authors of these studies should refrain from labelling them as “systematic review” and focus on ensuring that the methods used to generate the different outcomes and outcome domains are reported transparently.

Suggested Citation

  • Ewelina Rogozińska & Elizabeth Gargon & Rocío Olmedo-Requena & Amani Asour & Natalie A M Cooper & Claire L Vale & Janneke van’t Hooft, 2020. "Methods used to assess outcome consistency in clinical studies: A literature-based evaluation," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-12, July.
    • Handle: RePEc:plo:pone00:0235485
    DOI: 10.1371/journal.pone.0235485
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0235485
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0235485&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0235485?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kerry Dwan & Douglas G Altman & Juan A Arnaiz & Jill Bloom & An-Wen Chan & Eugenia Cronin & Evelyne Decullier & Philippa J Easterbrook & Erik Von Elm & Carrol Gamble & Davina Ghersi & John P A Ioannid, 2008. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias," PLOS ONE, Public Library of Science, vol. 3(8), pages 1-31, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Schaub, Jason & Stander, Willem J. & Montgomery, Paul, 2022. "LGBTQ+ Young People’s Health and Well-being Experiences in Out-of-home Social Care: A scoping review," Children and Youth Services Review, Elsevier, vol. 143(C).
    2. Paolo Crosetto & Antonio Filippin & Janna Heider, 2015. "A Study of Outcome Reporting Bias Using Gender Differences in Risk Attitudes," CESifo Economic Studies, CESifo Group, vol. 61(1), pages 239-262.
    3. Nikolaos Pandis & Padhraig S Fleming & Helen Worthington & Kerry Dwan & Georgia Salanti, 2015. "Discrepancies in Outcome Reporting Exist Between Protocols and Published Oral Health Cochrane Systematic Reviews," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-10, September.
    4. Wynanda A van Enst & Rob J P M Scholten & Lotty Hooft, 2012. "Identification of Additional Trials in Prospective Trial Registers for Cochrane Systematic Reviews," PLOS ONE, Public Library of Science, vol. 7(8), pages 1-5, August.
    5. Jamie J Kirkham & Doug G Altman & Paula R Williamson, 2010. "Bias Due to Changes in Specified Outcomes during the Systematic Review Process," PLOS ONE, Public Library of Science, vol. 5(3), pages 1-5, March.
    6. Daniele Fanelli, 2010. "Do Pressures to Publish Increase Scientists' Bias? An Empirical Support from US States Data," PLOS ONE, Public Library of Science, vol. 5(4), pages 1-7, April.
    7. Dongying Li & Tess Menotti & Yizhen Ding & Nancy M. Wells, 2021. "Life Course Nature Exposure and Mental Health Outcomes: A Systematic Review and Future Directions," IJERPH, MDPI, vol. 18(10), pages 1-28, May.
    8. Thomas Pfeiffer & Lars Bertram & John P A Ioannidis, 2011. "Quantifying Selective Reporting and the Proteus Phenomenon for Multiple Datasets with Similar Bias," PLOS ONE, Public Library of Science, vol. 6(3), pages 1-7, March.
    9. Ana Virgolino & Osvaldo Santos & Joana Costa & Mónica Fialho & Ivo Iavicoli & Tiina Santonen & Hanna Tolonen & Evangelia Samoli & Klea Katsouyanni & Georgios Baltatzis & Flavia Ruggieri & Annalisa Abb, 2021. "Challenges to Evidence Synthesis and Identification of Data Gaps in Human Biomonitoring," IJERPH, MDPI, vol. 18(6), pages 1-11, March.
    10. Michal Kicinski, 2013. "Publication Bias in Recent Meta-Analyses," PLOS ONE, Public Library of Science, vol. 8(11), pages 1-1, November.
    11. Carl Berning & Bernd Weiß, 2016. "Publication bias in the German social sciences: an application of the caliper test to three top-tier German social science journals," Quality & Quantity: International Journal of Methodology, Springer, vol. 50(2), pages 901-917, March.
    12. Thomas J. Kniesner & W. Kip Viscusi, 2023. "Compensating Differentials for Occupational Health and Safety Risks: Implications of Recent Evidence," Research in Labor Economics, in: 50th Celebratory Volume, volume 50, pages 83-116, Emerald Group Publishing Limited.
    13. Mangirdas Morkunas & Elzė Rudienė & Lukas Giriūnas & Laura Daučiūnienė, 2020. "Assessment of Factors Causing Bias in Marketing- Related Publications," Publications, MDPI, vol. 8(4), pages 1-16, October.
    14. Michał Krawczyk, 2015. "The Search for Significance: A Few Peculiarities in the Distribution of P Values in Experimental Psychology Literature," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-19, June.
    15. Violaine Smaïl-Faugeron & Hélène Fron Chabouis & Pierre Durieux & Jean-Pierre Attal & Michèle Muller-Bolla & Frédéric Courson, 2013. "Development of a Core Set of Outcomes for Randomized Controlled Trials with Multiple Outcomes – Example of Pulp Treatments of Primary Teeth for Extensive Decay in Children," PLOS ONE, Public Library of Science, vol. 8(1), pages 1-9, January.
    16. Robbie C M van Aert & Jelte M Wicherts & Marcel A L M van Assen, 2019. "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-32, April.
    17. Stephen A Gallo & Afton S Carpenter & David Irwin & Caitlin D McPartland & Joseph Travis & Sofie Reynders & Lisa A Thompson & Scott R Glisson, 2014. "The Validation of Peer Review through Research Impact Measures and the Implications for Funding Strategies," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-9, September.
    18. Jeffrey C. Valentine & Sandra Jo Wilson & David Rindskopf & Timothy S. Lau & Emily E. Tanner-Smith & Martha Yeide & Robin LaSota & Lisa Foster, 2017. "Synthesizing Evidence in Public Policy Contexts," Evaluation Review, , vol. 41(1), pages 3-26, February.
    19. Buehling, Kilian, 2021. "Changing research topic trends as an effect of publication rankings – The case of German economists and the Handelsblatt Ranking," Journal of Informetrics, Elsevier, vol. 15(3).
    20. Salandra, Rossella & Criscuolo, Paola & Salter, Ammon, 2021. "Directing scientists away from potentially biased publications: the role of systematic reviews in health care," Research Policy, Elsevier, vol. 50(1).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0235485. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.