IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v13y2022i1d10.1038_s41467-022-34646-2.html
   My bibliography  Save this article

Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients

Author

Listed:
  • Todd J. Levy

    (Feinstein Institutes for Medical Research, Northwell Health
    Feinstein Institutes for Medical Research, Northwell Health)

  • Kevin Coppa

    (Northwell Health)

  • Jinxuan Cang

    (Feinstein Institutes for Medical Research, Northwell Health
    Feinstein Institutes for Medical Research, Northwell Health)

  • Douglas P. Barnaby

    (Feinstein Institutes for Medical Research, Northwell Health
    Northwell Health)

  • Marc D. Paradis

    (Northwell Health)

  • Stuart L. Cohen

    (Feinstein Institutes for Medical Research, Northwell Health
    Northwell Health)

  • Alex Makhnevich

    (Feinstein Institutes for Medical Research, Northwell Health
    Northwell Health)

  • David Klaveren

    (Erasmus MC University Medical Center
    Tufts Medical Center)

  • David M. Kent

    (Tufts Medical Center)

  • Karina W. Davidson

    (Feinstein Institutes for Medical Research, Northwell Health
    Northwell Health)

  • Jamie S. Hirsch

    (Feinstein Institutes for Medical Research, Northwell Health
    Northwell Health
    Northwell Health)

  • Theodoros P. Zanos

    (Feinstein Institutes for Medical Research, Northwell Health
    Feinstein Institutes for Medical Research, Northwell Health
    Northwell Health)

Abstract

Clinical prognostic models can assist patient care decisions. However, their performance can drift over time and location, necessitating model monitoring and updating. Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for these drifts. We develop a framework for continuously monitoring and updating prognostic models and apply it to predict 28-day survival in COVID-19 patients. We use demographic, laboratory, and clinical data from electronic health records of 34912 hospitalized COVID-19 patients from March 2020 until May 2022 and compare three modeling methods. Model calibration performance drift is immediately detected with minor fluctuations in discrimination. The overall calibration on the prospective validation cohort is significantly improved when comparing the dynamically updated models against their static counterparts. Our findings suggest that, using this framework, models remain accurate and well-calibrated across various waves, variants, race and sex and yield positive net-benefits.

Suggested Citation

  • Todd J. Levy & Kevin Coppa & Jinxuan Cang & Douglas P. Barnaby & Marc D. Paradis & Stuart L. Cohen & Alex Makhnevich & David Klaveren & David M. Kent & Karina W. Davidson & Jamie S. Hirsch & Theodoros, 2022. "Development and validation of self-monitoring auto-updating prognostic models of survival for hospitalized COVID-19 patients," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    • Handle: RePEc:nat:natcom:v:13:y:2022:i:1:d:10.1038_s41467-022-34646-2
    DOI: 10.1038/s41467-022-34646-2
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-022-34646-2
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-022-34646-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Ewout W Steyerberg & Karel G M Moons & Danielle A van der Windt & Jill A Hayden & Pablo Perel & Sara Schroter & Richard D Riley & Harry Hemingway & Douglas G Altman & for the PROGRESS Group, 2013. "Prognosis Research Strategy (PROGRESS) 3: Prognostic Model Research," PLOS Medicine, Public Library of Science, vol. 10(2), pages 1-9, February.
    2. van Buuren, Stef & Groothuis-Oudshoorn, Karin, 2011. "mice: Multivariate Imputation by Chained Equations in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 45(i03).
    3. Xiaonan Zhang & Yun Tan & Yun Ling & Gang Lu & Feng Liu & Zhigang Yi & Xiaofang Jia & Min Wu & Bisheng Shi & Shuibao Xu & Jun Chen & Wei Wang & Bing Chen & Lu Jiang & Shuting Yu & Jing Lu & Jinzeng Wa, 2020. "Viral and host factors related to the clinical outcome of COVID-19," Nature, Nature, vol. 583(7816), pages 437-440, July.
    4. Vickers, Andrew J, 2008. "Decision Analysis for the Evaluation of Diagnostic Tests, Prediction Models, and Molecular Markers," The American Statistician, American Statistical Association, vol. 62(4), pages 314-320.
    5. Tyler H. McCormick & Adrian E. Raftery & David Madigan & Randall S. Burd, 2012. "Dynamic Logistic Regression and Dynamic Model Averaging for Binary Classification," Biometrics, The International Biometric Society, vol. 68(1), pages 23-30, March.
    6. Behzad Vahedi & Morteza Karimzadeh & Hamidreza Zoraghein, 2021. "Spatiotemporal prediction of COVID-19 cases using inter- and intra-county proxies of human interactions," Nature Communications, Nature, vol. 12(1), pages 1-15, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mélanie Roschewitz & Galvin Khara & Joe Yearsley & Nisha Sharma & Jonathan J. James & Éva Ambrózay & Adam Heroux & Peter Kecskemethy & Tobias Rijken & Ben Glocker, 2023. "Automatic correction of performance drift under acquisition shift in medical image classification," Nature Communications, Nature, vol. 14(1), pages 1-10, December.
    2. Ali Kore & Elyar Abbasi Bavil & Vallijah Subasri & Moustafa Abdalla & Benjamin Fine & Elham Dolatabadi & Mohamed Abdalla, 2024. "Empirical data drift detection experiments on real-world medical imaging data," Nature Communications, Nature, vol. 15(1), pages 1-11, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jiakun Jiang & Wei Yang & Erin M. Schnellinger & Stephen E. Kimmel & Wensheng Guo, 2023. "Dynamic logistic state space prediction model for clinical decision making," Biometrics, The International Biometric Society, vol. 79(1), pages 73-85, March.
    2. Noémi Kreif & Richard Grieve & Iván Díaz & David Harrison, 2015. "Evaluation of the Effect of a Continuous Treatment: A Machine Learning Approach with an Application to Treatment for Traumatic Brain Injury," Health Economics, John Wiley & Sons, Ltd., vol. 24(9), pages 1213-1228, September.
    3. Abhilash Bandam & Eedris Busari & Chloi Syranidou & Jochen Linssen & Detlef Stolten, 2022. "Classification of Building Types in Germany: A Data-Driven Modeling Approach," Data, MDPI, vol. 7(4), pages 1-23, April.
    4. Boonstra Philip S. & Little Roderick J.A. & West Brady T. & Andridge Rebecca R. & Alvarado-Leiton Fernanda, 2021. "A Simulation Study of Diagnostics for Selection Bias," Journal of Official Statistics, Sciendo, vol. 37(3), pages 751-769, September.
    5. Christopher J Greenwood & George J Youssef & Primrose Letcher & Jacqui A Macdonald & Lauryn J Hagg & Ann Sanson & Jenn Mcintosh & Delyse M Hutchinson & John W Toumbourou & Matthew Fuller-Tyszkiewicz &, 2020. "A comparison of penalised regression methods for informing the selection of predictive markers," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-14, November.
    6. Liangyuan Hu & Lihua Li, 2022. "Using Tree-Based Machine Learning for Health Studies: Literature Review and Case Series," IJERPH, MDPI, vol. 19(23), pages 1-13, December.
    7. Norah Alyabs & Sy Han Chiou, 2022. "The Missing Indicator Approach for Accelerated Failure Time Model with Covariates Subject to Limits of Detection," Stats, MDPI, vol. 5(2), pages 1-13, May.
    8. Feldkircher, Martin, 2014. "The determinants of vulnerability to the global financial crisis 2008 to 2009: Credit growth and other sources of risk," Journal of International Money and Finance, Elsevier, vol. 43(C), pages 19-49.
    9. Eunsil Seok & Akhgar Ghassabian & Yuyan Wang & Mengling Liu, 2024. "Statistical Methods for Modeling Exposure Variables Subject to Limit of Detection," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 16(2), pages 435-458, July.
    10. Ida Kubiszewski & Kenneth Mulder & Diane Jarvis & Robert Costanza, 2022. "Toward better measurement of sustainable development and wellbeing: A small number of SDG indicators reliably predict life satisfaction," Sustainable Development, John Wiley & Sons, Ltd., vol. 30(1), pages 139-148, February.
    11. Georges Steffgen & Philipp E. Sischka & Martha Fernandez de Henestrosa, 2020. "The Quality of Work Index and the Quality of Employment Index: A Multidimensional Approach of Job Quality and Its Links to Well-Being at Work," IJERPH, MDPI, vol. 17(21), pages 1-31, October.
    12. Christopher Kath & Florian Ziel, 2018. "The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts," Papers 1811.08604, arXiv.org.
    13. Alwin Schierenberg & Margaretha C Minnaard & Rogier M Hopstaken & Alma C van de Pol & Berna D L Broekhuizen & Niek J de Wit & Johannes B Reitsma & Saskia F van Vugt & Aleida W Graffelman & Hasse Melby, 2016. "External Validation of Prediction Models for Pneumonia in Primary Care Patients with Lower Respiratory Tract Infection: An Individual Patient Data Meta-Analysis," PLOS ONE, Public Library of Science, vol. 11(2), pages 1-16, February.
    14. Esef Hakan Toytok & Sungur Gürel, 2019. "Does Project Children’s University Increase Academic Self-Efficacy in 6th Graders? A Weak Experimental Design," Sustainability, MDPI, vol. 11(3), pages 1-12, February.
    15. J M van Niekerk & M C Vos & A Stein & L M A Braakman-Jansen & A F Voor in ‘t holt & J E W C van Gemert-Pijnen, 2020. "Risk factors for surgical site infections using a data-driven approach," PLOS ONE, Public Library of Science, vol. 15(10), pages 1-14, October.
    16. Catania, Leopoldo & Grassi, Stefano & Ravazzolo, Francesco, 2019. "Forecasting cryptocurrencies under model and parameter instability," International Journal of Forecasting, Elsevier, vol. 35(2), pages 485-501.
    17. Joost R. Ginkel, 2020. "Standardized Regression Coefficients and Newly Proposed Estimators for $${R}^{{2}}$$R2 in Multiply Imputed Data," Psychometrika, Springer;The Psychometric Society, vol. 85(1), pages 185-205, March.
    18. Lara Jehi & Xinge Ji & Alex Milinovich & Serpil Erzurum & Amy Merlino & Steve Gordon & James B Young & Michael W Kattan, 2020. "Development and validation of a model for individualized prediction of hospitalization risk in 4,536 patients with COVID-19," PLOS ONE, Public Library of Science, vol. 15(8), pages 1-15, August.
    19. Matthew Carli & Mary H. Ward & Catherine Metayer & David C. Wheeler, 2022. "Imputation of Below Detection Limit Missing Data in Chemical Mixture Analysis with Bayesian Group Index Regression," IJERPH, MDPI, vol. 19(3), pages 1-17, January.
    20. Gerko Vink & Laurence E. Frank & Jeroen Pannekoek & Stef Buuren, 2014. "Predictive mean matching imputation of semicontinuous variables," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 68(1), pages 61-90, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:13:y:2022:i:1:d:10.1038_s41467-022-34646-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.