IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v15y2024i1d10.1038_s41467-024-53450-8.html
   My bibliography  Save this article

An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Author

Listed:
  • Yuan Gao

    (Maastricht University Medical Centre
    Netherlands Cancer Institute
    Radboud University Medical Centre)

  • Sofia Ventura-Diaz

    (St Joseph’s Healthcare Hamilton)

  • Xin Wang

    (Maastricht University Medical Centre
    Netherlands Cancer Institute
    Radboud University Medical Centre)

  • Muzhen He

    (Shengli Clinical Medical College of Fujian Medical University, Fuzhou University Affiliated Provincial Hospital)

  • Zeyan Xu

    (The Third Affiliated Hospital of Kunming Medical University)

  • Arlene Weir

    (Cork University Hospital)

  • Hong-Yu Zhou

    (Harvard Medical School)

  • Tianyu Zhang

    (Maastricht University Medical Centre
    Netherlands Cancer Institute
    Radboud University Medical Centre)

  • Frederieke H. Duijnhoven

    (Netherlands Cancer Institute)

  • Luyi Han

    (Netherlands Cancer Institute
    Radboud University Medical Centre)

  • Xiaomei Li

    (The Second Clinical Medical College of Jinan University)

  • Anna D’Angelo

    (Oncological Radiotherapy and Hematology, Fondazione Policlinico Universitario ‘A. Gemelli’ IRCCS)

  • Valentina Longo

    (Oncological Radiotherapy and Hematology, Fondazione Policlinico Universitario ‘A. Gemelli’ IRCCS)

  • Zaiyi Liu

    (Southern Medical University
    Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application)

  • Jonas Teuwen

    (Netherlands Cancer Institute)

  • Marleen Kok

    (Netherlands Cancer Institute
    Netherlands Cancer Institute)

  • Regina Beets-Tan

    (Maastricht University Medical Centre
    Netherlands Cancer Institute)

  • Hugo M. Horlings

    (Netherlands Cancer Institute)

  • Tao Tan

    (Macao Polytechnic University)

  • Ritse Mann

    (Netherlands Cancer Institute
    Radboud University Medical Centre)

Abstract

Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP’s clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

Suggested Citation

  • Yuan Gao & Sofia Ventura-Diaz & Xin Wang & Muzhen He & Zeyan Xu & Arlene Weir & Hong-Yu Zhou & Tianyu Zhang & Frederieke H. Duijnhoven & Luyi Han & Xiaomei Li & Anna D’Angelo & Valentina Longo & Zaiyi, 2024. "An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
  • Handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-53450-8
    DOI: 10.1038/s41467-024-53450-8
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-024-53450-8
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1038/s41467-024-53450-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Andrew J. Vickers & Elena B. Elkin, 2006. "Decision Curve Analysis: A Novel Method for Evaluating Prediction Models," Medical Decision Making, , vol. 26(6), pages 565-574, November.
    2. Paolo Capogrosso & Andrew J. Vickers, 2019. "A Systematic Review of the Literature Demonstrates Some Errors in the Use of Decision Curve Analysis but Generally Correct Interpretation of Findings," Medical Decision Making, , vol. 39(5), pages 493-498, July.
    3. Cheng Jin & Heng Yu & Jia Ke & Peirong Ding & Yongju Yi & Xiaofeng Jiang & Xin Duan & Jinghua Tang & Daniel T. Chang & Xiaojian Wu & Feng Gao & Ruijiang Li, 2021. "Predicting treatment response from longitudinal images using multi-task deep learning," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    4. Stephen-John Sammut & Mireia Crispin-Ortuzar & Suet-Feung Chin & Elena Provenzano & Helen A. Bardwell & Wenxin Ma & Wei Cope & Ali Dariush & Sarah-Jane Dawson & Jean E. Abraham & Janet Dunn & Louise H, 2022. "Multi-omic machine learning predictor of breast cancer therapy response," Nature, Nature, vol. 601(7894), pages 623-629, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ja Hyeon Ku & Myong Kim & Seok-Soo Byun & Hyeon Jeong & Cheol Kwak & Hyeon Hoe Kim & Sang Eun Lee, 2015. "External Validation of Models for Prediction of Lymph Node Metastasis in Urothelial Carcinoma of the Bladder," PLOS ONE, Public Library of Science, vol. 10(10), pages 1-10, October.
    2. Lin Lu & Laurent Dercle & Binsheng Zhao & Lawrence H. Schwartz, 2021. "Deep learning for the prediction of early on-treatment response in metastatic colorectal cancer from serial medical imaging," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    3. Yiwang Zhou & Peter X.K. Song & Haoda Fu, 2021. "Net benefit index: Assessing the influence of a biomarker for individualized treatment rules," Biometrics, The International Biometric Society, vol. 77(4), pages 1254-1264, December.
    4. Jing Sun & Yue Liu & Jianhui Zhao & Bin Lu & Siyun Zhou & Wei Lu & Jingsun Wei & Yeting Hu & Xiangxing Kong & Junshun Gao & Hong Guan & Junli Gao & Qian Xiao & Xue Li, 2024. "Plasma proteomic and polygenic profiling improve risk stratification and personalized screening for colorectal cancer," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    5. Shamil D. Cooray & Lihini A. Wijeyaratne & Georgia Soldatos & John Allotey & Jacqueline A. Boyle & Helena J. Teede, 2020. "The Unrealised Potential for Predicting Pregnancy Complications in Women with Gestational Diabetes: A Systematic Review and Critical Appraisal," IJERPH, MDPI, vol. 17(9), pages 1-20, April.
    6. Khushal Arjan & Lui G Forni & Richard M Venn & David Hunt & Luke Eliot Hodgson, 2021. "Clinical decision-making in older adults following emergency admission to hospital. Derivation and validation of a risk stratification score: OPERA," PLOS ONE, Public Library of Science, vol. 16(3), pages 1-12, March.
    7. Christian Bock & Joan Elias Walter & Bastian Rieck & Ivo Strebel & Klara Rumora & Ibrahim Schaefer & Michael J. Zellweger & Karsten Borgwardt & Christian Müller, 2024. "Enhancing the diagnosis of functionally relevant coronary artery disease with machine learning," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    8. Tracey L. Marsh & Holly Janes & Margaret S. Pepe, 2020. "Statistical inference for net benefit measures in biomarker validation studies," Biometrics, The International Biometric Society, vol. 76(3), pages 843-852, September.
    9. Kathleen F. Kerr & Tracey L. Marsh & Holly Janes, 2019. "The Importance of Uncertainty and Opt-In v. Opt-Out: Best Practices for Decision Curve Analysis," Medical Decision Making, , vol. 39(5), pages 491-492, July.
    10. Chang Wook Jeong & Sangchul Lee & Jin-Woo Jung & Byung Ki Lee & Seong Jin Jeong & Sung Kyu Hong & Seok-Soo Byun & Sang Eun Lee, 2014. "Mobile Application-Based Seoul National University Prostate Cancer Risk Calculator: Development, Validation, and Comparative Analysis with Two Western Risk Calculators in Korean Men," PLOS ONE, Public Library of Science, vol. 9(4), pages 1-7, April.
    11. Yanqing Wang & Yingqi Zhao & Yingye Zheng, 2022. "Targeted Search for Individualized Clinical Decision Rules to Optimize Clinical Outcomes," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 14(3), pages 564-581, December.
    12. Tae Yoon Lee & Paul Gustafson & Mohsen Sadatsafavi, 2023. "Closed-Form Solution of the Unit Normal Loss Integral in 2 Dimensions, with Application in Value-of-Information Analysis," Medical Decision Making, , vol. 43(5), pages 621-626, July.
    13. Baker Stuart G. & Van Calster Ben & Steyerberg Ewout W., 2012. "Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update," The International Journal of Biostatistics, De Gruyter, vol. 8(1), pages 1-37, March.
    14. Kevin Sandeman & Juho T Eineluoto & Joona Pohjonen & Andrew Erickson & Tuomas P Kilpeläinen & Petrus Järvinen & Henrikki Santti & Anssi Petas & Mika Matikainen & Suvi Marjasuo & Anu Kenttämies & Tuoma, 2020. "Prostate MRI added to CAPRA, MSKCC and Partin cancer nomograms significantly enhances the prediction of adverse findings and biochemical recurrence after radical prostatectomy," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-14, July.
    15. Danny J N Wong & Steve Harris & Arun Sahni & James R Bedford & Laura Cortes & Richard Shawyer & Andrew M Wilson & Helen A Lindsay & Doug Campbell & Scott Popham & Lisa M Barneto & Paul S Myles & SNAP-, 2020. "Developing and validating subjective and objective risk-assessment measures for predicting mortality after major surgery: An international prospective cohort study," PLOS Medicine, Public Library of Science, vol. 17(10), pages 1-22, October.
    16. James M. Dolezal & Andrew Srisuwananukorn & Dmitry Karpeyev & Siddhi Ramesh & Sara Kochanny & Brittany Cody & Aaron S. Mansfield & Sagar Rakshit & Radhika Bansal & Melanie C. Bois & Aaron O. Bungum & , 2022. "Uncertainty-informed deep learning models enable high-confidence predictions for digital histopathology," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    17. François Luthi & Olivier Deriaz & Philippe Vuistiner & Cyrille Burrus & Roger Hilfiker, 2014. "Predicting Non Return to Work after Orthopaedic Trauma: The Wallis Occupational Rehabilitation RisK (WORRK) Model," PLOS ONE, Public Library of Science, vol. 9(4), pages 1-11, April.
    18. Capponi, Giovanna & Martinelli, Arianna & Nuvolari, Alessandro, 2022. "Breakthrough innovations and where to find them," Research Policy, Elsevier, vol. 51(1).
    19. Seungmin Lee & Jeong Soo Park & Hyowon Woo & Yong Kyoung Yoo & Dongho Lee & Seok Chung & Dae Sung Yoon & Ki- Baek Lee & Jeong Hoon Lee, 2024. "Rapid deep learning-assisted predictive diagnostics for point-of-care testing," Nature Communications, Nature, vol. 15(1), pages 1-12, December.
    20. Ying Huang & Eric Laber, 2016. "Personalized Evaluation of Biomarker Value: A Cost-Benefit Perspective," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 8(1), pages 43-65, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:15:y:2024:i:1:d:10.1038_s41467-024-53450-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.