IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-45355-3.html
   My bibliography  Save this article

Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines

Author

Listed:
  • Alexander P. L. Martindale

    (Brighton and Sussex Medical School)

  • Carrie D. Llewellyn

    (Brighton and Sussex Medical School)

  • Richard O. Visser

    (Brighton and Sussex Medical School)

  • Benjamin Ng

    (Sandwell and West Birmingham NHS Trust
    University of Oxford)

  • Victoria Ngai

    (University College London Medical School)

  • Aditya U. Kale

    (University of Birmingham
    University Hospitals Birmingham NHS Foundation Trust
    University of Birmingham)

  • Lavinia Ferrante Ruffano

    (University of York)

  • Robert M. Golub

    (Northwestern University Feinberg School of Medicine)

  • Gary S. Collins

    (University of Oxford)

  • David Moher

    (Ottawa Hospital Research Institute)

  • Melissa D. McCradden

    (The Hospital for Sick Children
    Peter Gilgan Centre for Research & Learning
    Dalla Lana School of Public Health)

  • Lauren Oakden-Rayner

    (University of Adelaide)

  • Samantha Cruz Rivera

    (University of Birmingham
    University of Birmingham)

  • Melanie Calvert

    (University of Birmingham
    University of Birmingham
    University of Birmingham
    University of Birmingham)

  • Christopher J. Kelly

    (Google Health)

  • Cecilia S. Lee

    (University of Washington)

  • Christopher Yau

    (University of Oxford
    Health Data Research UK)

  • An-Wen Chan

    (Women’s College Hospital. University of Toronto)

  • Pearse A. Keane

    (Moorfields Eye Hospital NHS Foundation Trust and UCL Institute of Ophthalmology)

  • Andrew L. Beam

    (Harvard. T.H. Chan School of Public Health
    Harvard Medical School)

  • Alastair K. Denniston

    (University of Birmingham
    University Hospitals Birmingham NHS Foundation Trust
    University of Birmingham
    University of Birmingham)

  • Xiaoxuan Liu

    (University of Birmingham
    University Hospitals Birmingham NHS Foundation Trust
    University of Birmingham)

Abstract

The Consolidated Standards of Reporting Trials extension for Artificial Intelligence interventions (CONSORT-AI) was published in September 2020. Since its publication, several randomised controlled trials (RCTs) of AI interventions have been published but their completeness and transparency of reporting is unknown. This systematic review assesses the completeness of reporting of AI RCTs following publication of CONSORT-AI and provides a comprehensive summary of RCTs published in recent years. 65 RCTs were identified, mostly conducted in China (37%) and USA (18%). Median concordance with CONSORT-AI reporting was 90% (IQR 77–94%), although only 10 RCTs explicitly reported its use. Several items were consistently under-reported, including algorithm version, accessibility of the AI intervention or code, and references to a study protocol. Only 3 of 52 included journals explicitly endorsed or mandated CONSORT-AI. Despite a generally high concordance amongst recent AI RCTs, some AI-specific considerations remain systematically poorly reported. Further encouragement of CONSORT-AI adoption by journals and funders may enable more complete adoption of the full CONSORT-AI guidelines.

Suggested Citation

  • Alexander P. L. Martindale & Carrie D. Llewellyn & Richard O. Visser & Benjamin Ng & Victoria Ngai & Aditya U. Kale & Lavinia Ferrante Ruffano & Robert M. Golub & Gary S. Collins & David Moher & Melis, 2024. "Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-45355-3
    DOI: 10.1038/s41467-024-45355-3
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-45355-3
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-45355-3?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    2. Scott Mayer McKinney & Marcin Sieniek & Varun Godbole & Jonathan Godwin & Natasha Antropova & Hutan Ashrafian & Trevor Back & Mary Chesus & Greg S. Corrado & Ara Darzi & Mozziyar Etemadi & Florencia G, 2020. "International evaluation of an AI system for breast cancer screening," Nature, Nature, vol. 577(7788), pages 89-94, January.
    3. Peter MacPherson & Emily L Webb & Wala Kamchedzera & Elizabeth Joekes & Gugu Mjoli & David G Lalloo & Titus H Divala & Augustine T Choko & Rachael M Burke & Hendramoorthy Maheswaran & Madhukar Pai & S, 2021. "Computer-aided X-ray screening for tuberculosis and HIV testing among adults with cough in Malawi (the PROSPECT study): A randomised trial and cost-effectiveness analysis," PLOS Medicine, Public Library of Science, vol. 18(9), pages 1-17, September.
    4. Matthew J Page & Joanne E McKenzie & Patrick M Bossuyt & Isabelle Boutron & Tammy C Hoffmann & Cynthia D Mulrow & Larissa Shamseer & Jennifer M Tetzlaff & Elie A Akl & Sue E Brennan & Roger Chou & Jul, 2021. "The PRISMA 2020 statement: An updated guideline for reporting systematic reviews," PLOS Medicine, Public Library of Science, vol. 18(3), pages 1-15, March.
    5. Scott Mayer McKinney & Marcin Sieniek & Varun Godbole & Jonathan Godwin & Natasha Antropova & Hutan Ashrafian & Trevor Back & Mary Chesus & Greg S. Corrado & Ara Darzi & Mozziyar Etemadi & Florencia G, 2020. "Addendum: International evaluation of an AI system for breast cancer screening," Nature, Nature, vol. 586(7829), pages 19-19, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joachim Meyer, 2024. "Doing AI: Algorithmic decision support as a human activity," Papers 2402.14674, arXiv.org, revised Apr 2024.
    2. Babak Abedin & Christian Meske & Iris Junglas & Fethi Rabhi & Hamid R. Motahari-Nezhad, 2022. "Designing and Managing Human-AI Interactions," Information Systems Frontiers, Springer, vol. 24(3), pages 691-697, June.
    3. Armando Vargas-Palacios & Nisha Sharma & Gurdeep S. Sagoo, 2023. "Cost-effectiveness requirements for implementing artificial intelligence technology in the Women’s UK Breast Cancer Screening service," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    4. Yuming Jiang & Zhicheng Zhang & Wei Wang & Weicai Huang & Chuanli Chen & Sujuan Xi & M. Usman Ahmad & Yulan Ren & Shengtian Sang & Jingjing Xie & Jen-Yeu Wang & Wenjun Xiong & Tuanjie Li & Zhen Han & , 2023. "Biology-guided deep learning predicts prognosis and cancer immunotherapy response," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    5. Shu Jiang & Jiguo Cao & Bernard Rosner & Graham A. Colditz, 2023. "Supervised two‐dimensional functional principal component analysis with time‐to‐event outcomes and mammogram imaging data," Biometrics, The International Biometric Society, vol. 79(2), pages 1359-1369, June.
    6. Helen M. L. Frazer & Carlos A. Peña-Solorzano & Chun Fung Kwok & Michael S. Elliott & Yuanhong Chen & Chong Wang & Jocelyn F. Lippey & John L. Hopper & Peter Brotchie & Gustavo Carneiro & Davis J. McC, 2024. "Comparison of AI-integrated pathways with human-AI interaction in population mammographic screening for breast cancer," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    7. Minkyu Shin & Jin Kim & Bas van Opheusden & Thomas L. Griffiths, 2023. "Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Novelty," Papers 2303.07462, arXiv.org, revised Apr 2023.
    8. Juexiao Zhou & Haoyang Li & Xingyu Liao & Bin Zhang & Wenjia He & Zhongxiao Li & Longxi Zhou & Xin Gao, 2023. "A unified method to revoke the private data of patients in intelligent healthcare with audit to forget," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    9. Sebastian Schleidgen & Orsolya Friedrich & Selin Gerlek & Galia Assadi & Johanna Seifert, 2023. "The concept of “interaction” in debates on human–machine interaction," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-13, December.
    10. Taynara de Oliveira Castellões & Paloma Maria Silva Rocha Rizol & Luiz Fernando Costa Nascimento, 2024. "Association between Premature Birth and Air Pollutants Using Fuzzy and Adaptive Neuro-Fuzzy Inference System (ANFIS) Techniques," Mathematics, MDPI, vol. 12(18), pages 1-12, September.
    11. Qianwei Zhou & Margarita Zuley & Yuan Guo & Lu Yang & Bronwyn Nair & Adrienne Vargo & Suzanne Ghannam & Dooman Arefan & Shandong Wu, 2021. "A machine and human reader study on AI diagnosis model safety under attacks of adversarial images," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    12. Mélanie Roschewitz & Galvin Khara & Joe Yearsley & Nisha Sharma & Jonathan J. James & Éva Ambrózay & Adam Heroux & Peter Kecskemethy & Tobias Rijken & Ben Glocker, 2023. "Automatic correction of performance drift under acquisition shift in medical image classification," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    13. Michael J. Fell & Alexandra Schneiders & David Shipworth, 2019. "Consumer Demand for Blockchain-Enabled Peer-to-Peer Electricity Trading in the United Kingdom: An Online Survey Experiment," Energies, MDPI, vol. 12(20), pages 1-25, October.
    14. Su Keng Tan & Wai Keung Leung & Alexander Tin Hong Tang & Roger A Zwahlen, 2017. "Effects of mandibular setback with or without maxillary advancement osteotomies on pharyngeal airways: An overview of systematic reviews," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-20, October.
    15. Rizkiana Sidqiyatul Hamdani & Sudharto Prawata Hadi & Iwan Rudiarto, 2021. "Progress or Regress? A Systematic Review on Two Decades of Monitoring and Addressing Land Subsidence Hazards in Semarang City," Sustainability, MDPI, vol. 13(24), pages 1-26, December.
    16. Boglárka Anna Éliás & Attila Jámbor, 2021. "Food Security and COVID-19: A Systematic Review of the First-Year Experience," Sustainability, MDPI, vol. 13(9), pages 1-18, May.
    17. Ángel Enrique & Juana Bretón-López & Guadalupe Molinari & Rosa M. Baños & Cristina Botella, 2018. "Efficacy of an adaptation of the Best Possible Self intervention implemented through positive technology: a randomized control trial," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 13(3), pages 671-689, September.
    18. Omoyele, Olalekan & Hoffmann, Maximilian & Koivisto, Matti & Larrañeta, Miguel & Weinand, Jann Michael & Linßen, Jochen & Stolten, Detlef, 2024. "Increasing the resolution of solar and wind time series for energy system modeling: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    19. Gerben ter Riet & Paula Chesley & Alan G Gross & Lara Siebeling & Patrick Muggensturm & Nadine Heller & Martin Umbehr & Daniela Vollenweider & Tsung Yu & Elie A Akl & Lizzy Brewster & Olaf M Dekkers &, 2013. "All That Glitters Isn't Gold: A Survey on Acknowledgment of Limitations in Biomedical Studies," PLOS ONE, Public Library of Science, vol. 8(11), pages 1-6, November.
    20. Iranzu Mugueta-Aguinaga & Begonya Garcia-Zapirain, 2017. "FRED: Exergame to Prevent Dependence and Functional Deterioration Associated with Ageing. A Pilot Three-Week Randomized Controlled Clinical Trial," IJERPH, MDPI, vol. 14(12), pages 1-18, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-45355-3. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.