IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-46000-9.html
   My bibliography  Save this article

A framework for evaluating clinical artificial intelligence systems without ground-truth annotations

Author

Listed:
  • Dani Kiyasseh

    (Cedars-Sinai Medical Center)

  • Aaron Cohen

    (Flatiron Health
    New York University School of Medicine)

  • Chengsheng Jiang

    (Flatiron Health)

  • Nicholas Altieri

    (Flatiron Health)

Abstract

A clinical artificial intelligence (AI) system is often validated on data withheld during its development. This provides an estimate of its performance upon future deployment on data in the wild; those currently unseen but are expected to be encountered in a clinical setting. However, estimating performance on data in the wild is complicated by distribution shift between data in the wild and withheld data and the absence of ground-truth annotations. Here, we introduce SUDO, a framework for evaluating AI systems on data in the wild. Through experiments on AI systems developed for dermatology images, histopathology patches, and clinical notes, we show that SUDO can identify unreliable predictions, inform the selection of models, and allow for the previously out-of-reach assessment of algorithmic bias for data in the wild without ground-truth annotations. These capabilities can contribute to the deployment of trustworthy and ethical AI systems in medicine.

Suggested Citation

  • Dani Kiyasseh & Aaron Cohen & Chengsheng Jiang & Nicholas Altieri, 2024. "A framework for evaluating clinical artificial intelligence systems without ground-truth annotations," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-46000-9
    DOI: 10.1038/s41467-024-46000-9
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-46000-9
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-024-46000-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Andre Esteva & Brett Kuprel & Roberto A. Novoa & Justin Ko & Susan M. Swetter & Helen M. Blau & Sebastian Thrun, 2017. "Correction: Corrigendum: Dermatologist-level classification of skin cancer with deep neural networks," Nature, Nature, vol. 546(7660), pages 686-686, June.
    2. Chinyereugo M Umemneku Chikere & Kevin Wilson & Sara Graziadio & Luke Vale & A Joy Allen, 2019. "Diagnostic test evaluation methodology: A systematic review of methods employed to evaluate diagnostic tests in the absence of gold standard – An update," PLOS ONE, Public Library of Science, vol. 14(10), pages 1-25, October.
    3. Jessica L. Gronsbell & Tianxi Cai, 2018. "Semi‐supervised approaches to efficient evaluation of model prediction performance," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 80(3), pages 579-594, June.
    4. James M. Dolezal & Andrew Srisuwananukorn & Dmitry Karpeyev & Siddhi Ramesh & Sara Kochanny & Brittany Cody & Aaron S. Mansfield & Sagar Rakshit & Radhika Bansal & Melanie C. Bois & Aaron O. Bungum & , 2022. "Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    5. Andre Esteva & Brett Kuprel & Roberto A. Novoa & Justin Ko & Susan M. Swetter & Helen M. Blau & Sebastian Thrun, 2017. "Dermatologist-level classification of skin cancer with deep neural networks," Nature, Nature, vol. 542(7639), pages 115-118, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lin Lu & Laurent Dercle & Binsheng Zhao & Lawrence H. Schwartz, 2021. "Deep learning for the prediction of early on-treatment response in metastatic colorectal cancer from serial medical imaging," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    2. Zheng Yan & Wenqian Robertson & Yaosheng Lou & Tom W. Robertson & Sung Yong Park, 2021. "Finding leading scholars in mobile phone behavior: a mixed-method analysis of an emerging interdisciplinary field," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9499-9517, December.
    3. Freddy Gabbay & Rotem Lev Aharoni & Ori Schweitzer, 2022. "Deep Neural Network Memory Performance and Throughput Modeling and Simulation Framework," Mathematics, MDPI, vol. 10(21), pages 1-20, November.
    4. Gang Yu & Kai Sun & Chao Xu & Xing-Hua Shi & Chong Wu & Ting Xie & Run-Qi Meng & Xiang-He Meng & Kuan-Song Wang & Hong-Mei Xiao & Hong-Wen Deng, 2021. "Accurate recognition of colorectal cancer with semi-supervised deep learning on pathological images," Nature Communications, Nature, vol. 12(1), pages 1-13, December.
    5. DonHee Lee & Seong No Yoon, 2021. "Application of Artificial Intelligence-Based Technologies in the Healthcare Industry: Opportunities and Challenges," IJERPH, MDPI, vol. 18(1), pages 1-18, January.
    6. Dario Sipari & Betsy D. M. Chaparro-Rico & Daniele Cafolla, 2022. "SANE (Easy Gait Analysis System): Towards an AI-Assisted Automatic Gait-Analysis," IJERPH, MDPI, vol. 19(16), pages 1-27, August.
    7. Julian Schiele & Thomas Koperna & Jens O. Brunner, 2021. "Predicting intensive care unit bed occupancy for integrated operating room scheduling via neural networks," Naval Research Logistics (NRL), John Wiley & Sons, vol. 68(1), pages 65-88, February.
    8. Oded Rotem & Tamar Schwartz & Ron Maor & Yishay Tauber & Maya Tsarfati Shapiro & Marcos Meseguer & Daniella Gilboa & Daniel S. Seidman & Assaf Zaritsky, 2024. "Visual interpretability of image-based classification models by generative latent space disentanglement applied to in vitro fertilization," Nature Communications, Nature, vol. 15(1), pages 1-19, December.
    9. Taneja, Anu & Arora, Anuja, 2019. "Modeling user preferences using neural networks and tensor factorization model," International Journal of Information Management, Elsevier, vol. 45(C), pages 132-148.
    10. Hanning Ying & Xiaoqing Liu & Min Zhang & Yiyue Ren & Shihui Zhen & Xiaojie Wang & Bo Liu & Peng Hu & Lian Duan & Mingzhi Cai & Ming Jiang & Xiangdong Cheng & Xiangyang Gong & Haitao Jiang & Jianshuai, 2024. "A multicenter clinical AI system study for detection and diagnosis of focal liver lesions," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    11. Cristian Simionescu & Adrian Iftene, 2022. "Deep Learning Research Directions in Medical Imaging," Mathematics, MDPI, vol. 10(23), pages 1-25, November.
    12. Jingui Zhang & Chuangji Meng & Cunlu Xu & Jingyong Ma & Wei Su, 2022. "Deep Transfer Learning Method Based on Automatic Domain Alignment and Moment Matching," Mathematics, MDPI, vol. 10(14), pages 1-14, July.
    13. Yuming Jiang & Zhicheng Zhang & Wei Wang & Weicai Huang & Chuanli Chen & Sujuan Xi & M. Usman Ahmad & Yulan Ren & Shengtian Sang & Jingjing Xie & Jen-Yeu Wang & Wenjun Xiong & Tuanjie Li & Zhen Han & , 2023. "Biology-guided deep learning predicts prognosis and cancer immunotherapy response," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    14. Marta Mazur & Artnora Ndokaj & Divyambika Catakapatri Venugopal & Michela Roberto & Cristina Albu & Maciej Jedliński & Silverio Tomao & Iole Vozza & Grzegorz Trybek & Livia Ottolenghi & Fabrizio Guerr, 2021. "In Vivo Imaging-Based Techniques for Early Diagnosis of Oral Potentially Malignant Disorders—Systematic Review and Meta-Analysis," IJERPH, MDPI, vol. 18(22), pages 1-22, November.
    15. Khalid A. Ibrahim & Kristin S. Grußmayer & Nathan Riguet & Lely Feletti & Hilal A. Lashuel & Aleksandra Radenovic, 2023. "Label-free identification of protein aggregates using deep learning," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    16. Songhee Cheon & Jungyoon Kim & Jihye Lim, 2019. "The Use of Deep Learning to Predict Stroke Patient Mortality," IJERPH, MDPI, vol. 16(11), pages 1-12, May.
    17. Hailong He & Christine Schönmann & Mathias Schwarz & Benedikt Hindelang & Andrei Berezhnoi & Susanne Annette Steimle-Grauer & Ulf Darsow & Juan Aguirre & Vasilis Ntziachristos, 2022. "Fast raster-scan optoacoustic mesoscopy enables assessment of human melanoma microvasculature in vivo," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    18. Zilong Zhou & Hang Yuan & Xin Cai, 2023. "Rock Thin Section Image Identification Based on Convolutional Neural Networks of Adaptive and Second-Order Pooling Methods," Mathematics, MDPI, vol. 11(5), pages 1-27, March.
    19. Marios Constantinou & Themis Exarchos & Aristidis G. Vrahatis & Panagiotis Vlamos, 2023. "COVID-19 Classification on Chest X-ray Images Using Deep Learning Methods," IJERPH, MDPI, vol. 20(3), pages 1-13, January.
    20. Yun-Tsan Chang & Pacôme Prompsy & Susanne Kimeswenger & Yi-Chien Tsai & Desislava Ignatova & Olesya Pavlova & Christoph Iselin & Lars E. French & Mitchell P. Levesque & François Kuonen & Malgorzata Bo, 2024. "MHC-I upregulation safeguards neoplastic T cells in the skin against NK cell-mediated eradication in mycosis fungoides," Nature Communications, Nature, vol. 15(1), pages 1-18, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-46000-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.