IDEAS home Printed from https://ideas.repec.org/a/nat/nature/v577y2020i7788d10.1038_s41586-019-1799-6.html
   My bibliography  Save this article

International evaluation of an AI system for breast cancer screening

Author

Listed:
  • Scott Mayer McKinney

    (Google Health)

  • Marcin Sieniek

    (Google Health)

  • Varun Godbole

    (Google Health)

  • Jonathan Godwin

    (DeepMind)

  • Natasha Antropova

    (DeepMind)

  • Hutan Ashrafian

    (Imperial College London
    Imperial College London)

  • Trevor Back

    (DeepMind)

  • Mary Chesus

    (DeepMind)

  • Greg S. Corrado

    (Google Health)

  • Ara Darzi

    (Imperial College London
    Imperial College London
    Imperial College London)

  • Mozziyar Etemadi

    (Northwestern Medicine)

  • Florencia Garcia-Vicente

    (Northwestern Medicine)

  • Fiona J. Gilbert

    (University of Cambridge)

  • Mark Halling-Brown

    (Royal Surrey County Hospital)

  • Demis Hassabis

    (DeepMind)

  • Sunny Jansen

    (Verily Life Sciences)

  • Alan Karthikesalingam

    (Google Health)

  • Christopher J. Kelly

    (Google Health)

  • Dominic King

    (Google Health)

  • Joseph R. Ledsam

    (DeepMind)

  • David Melnick

    (Northwestern Medicine)

  • Hormuz Mostofi

    (Google Health)

  • Lily Peng

    (Google Health)

  • Joshua Jay Reicher

    (Stanford Health Care and Palo Alto Veterans Affairs)

  • Bernardino Romera-Paredes

    (DeepMind)

  • Richard Sidebottom

    (The Royal Marsden Hospital
    Thirlestaine Breast Centre)

  • Mustafa Suleyman

    (DeepMind)

  • Daniel Tse

    (Google Health)

  • Kenneth C. Young

    (Royal Surrey County Hospital)

  • Jeffrey Fauw

    (DeepMind)

  • Shravya Shetty

    (Google Health)

Abstract

Screening mammography aims to identify breast cancer at earlier stages of the disease, when treatment can be more successful1. Despite the existence of screening programmes worldwide, the interpretation of mammograms is affected by high rates of false positives and false negatives2. Here we present an artificial intelligence (AI) system that is capable of surpassing human experts in breast cancer prediction. To assess its performance in the clinical setting, we curated a large representative dataset from the UK and a large enriched dataset from the USA. We show an absolute reduction of 5.7% and 1.2% (USA and UK) in false positives and 9.4% and 2.7% in false negatives. We provide evidence of the ability of the system to generalize from the UK to the USA. In an independent study of six radiologists, the AI system outperformed all of the human readers: the area under the receiver operating characteristic curve (AUC-ROC) for the AI system was greater than the AUC-ROC for the average radiologist by an absolute margin of 11.5%. We ran a simulation in which the AI system participated in the double-reading process that is used in the UK, and found that the AI system maintained non-inferior performance and reduced the workload of the second reader by 88%. This robust assessment of the AI system paves the way for clinical trials to improve the accuracy and efficiency of breast cancer screening.

Suggested Citation

  • Scott Mayer McKinney & Marcin Sieniek & Varun Godbole & Jonathan Godwin & Natasha Antropova & Hutan Ashrafian & Trevor Back & Mary Chesus & Greg S. Corrado & Ara Darzi & Mozziyar Etemadi & Florencia G, 2020. "International evaluation of an AI system for breast cancer screening," Nature, Nature, vol. 577(7788), pages 89-94, January.
    • Handle: RePEc:nat:nature:v:577:y:2020:i:7788:d:10.1038_s41586-019-1799-6
    DOI: 10.1038/s41586-019-1799-6
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41586-019-1799-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1038/s41586-019-1799-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alexander P. L. Martindale & Carrie D. Llewellyn & Richard O. Visser & Benjamin Ng & Victoria Ngai & Aditya U. Kale & Lavinia Ferrante Ruffano & Robert M. Golub & Gary S. Collins & David Moher & Melis, 2024. "Concordance of randomised controlled trials for artificial intelligence interventions with the CONSORT-AI reporting guidelines," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    2. Minkyu Shin & Jin Kim & Bas van Opheusden & Thomas L. Griffiths, 2023. "Superhuman Artificial Intelligence Can Improve Human Decision Making by Increasing Novelty," Papers 2303.07462, arXiv.org, revised Apr 2023.
    3. Sebastian Schleidgen & Orsolya Friedrich & Selin Gerlek & Galia Assadi & Johanna Seifert, 2023. "The concept of “interaction” in debates on human–machine interaction," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-13, December.
    4. Qianwei Zhou & Margarita Zuley & Yuan Guo & Lu Yang & Bronwyn Nair & Adrienne Vargo & Suzanne Ghannam & Dooman Arefan & Shandong Wu, 2021. "A machine and human reader study on AI diagnosis model safety under attacks of adversarial images," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    5. Juexiao Zhou & Haoyang Li & Xingyu Liao & Bin Zhang & Wenjia He & Zhongxiao Li & Longxi Zhou & Xin Gao, 2023. "A unified method to revoke the private data of patients in intelligent healthcare with audit to forget," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    6. Armando Vargas-Palacios & Nisha Sharma & Gurdeep S. Sagoo, 2023. "Cost-effectiveness requirements for implementing artificial intelligence technology in the Women’s UK Breast Cancer Screening service," Nature Communications, Nature, vol. 14(1), pages 1-11, December.
    7. Helen M. L. Frazer & Carlos A. Peña-Solorzano & Chun Fung Kwok & Michael S. Elliott & Yuanhong Chen & Chong Wang & Jocelyn F. Lippey & John L. Hopper & Peter Brotchie & Gustavo Carneiro & Davis J. McC, 2024. "Comparison of AI-integrated pathways with human-AI interaction in population mammographic screening for breast cancer," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    8. Joachim Meyer, 2024. "Doing AI: Algorithmic decision support as a human activity," Papers 2402.14674, arXiv.org, revised Apr 2024.
    9. Mélanie Roschewitz & Galvin Khara & Joe Yearsley & Nisha Sharma & Jonathan J. James & Éva Ambrózay & Adam Heroux & Peter Kecskemethy & Tobias Rijken & Ben Glocker, 2023. "Automatic correction of performance drift under acquisition shift in medical image classification," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    10. Yuming Jiang & Zhicheng Zhang & Wei Wang & Weicai Huang & Chuanli Chen & Sujuan Xi & M. Usman Ahmad & Yulan Ren & Shengtian Sang & Jingjing Xie & Jen-Yeu Wang & Wenjun Xiong & Tuanjie Li & Zhen Han & , 2023. "Biology-guided deep learning predicts prognosis and cancer immunotherapy response," Nature Communications, Nature, vol. 14(1), pages 1-16, December.
    11. Taynara de Oliveira Castellões & Paloma Maria Silva Rocha Rizol & Luiz Fernando Costa Nascimento, 2024. "Association between Premature Birth and Air Pollutants Using Fuzzy and Adaptive Neuro-Fuzzy Inference System (ANFIS) Techniques," Mathematics, MDPI, vol. 12(18), pages 1-12, September.
    12. Babak Abedin & Christian Meske & Iris Junglas & Fethi Rabhi & Hamid R. Motahari-Nezhad, 2022. "Designing and Managing Human-AI Interactions," Information Systems Frontiers, Springer, vol. 24(3), pages 691-697, June.
    13. Shu Jiang & Jiguo Cao & Bernard Rosner & Graham A. Colditz, 2023. "Supervised two‐dimensional functional principal component analysis with time‐to‐event outcomes and mammogram imaging data," Biometrics, The International Biometric Society, vol. 79(2), pages 1359-1369, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:nature:v:577:y:2020:i:7788:d:10.1038_s41586-019-1799-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.