IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i20p4246-d1257599.html
   My bibliography  Save this article

Sensitivity of Survival Analysis Metrics

Author

Listed:
  • Iulii Vasilev

    (Computer Science Department, Lomonosov Moscow State University, Vorobjovy Gory, 119899 Moscow, Russia)

  • Mikhail Petrovskiy

    (Computer Science Department, Lomonosov Moscow State University, Vorobjovy Gory, 119899 Moscow, Russia)

  • Igor Mashechkin

    (Computer Science Department, Lomonosov Moscow State University, Vorobjovy Gory, 119899 Moscow, Russia)

Abstract

Survival analysis models allow for predicting the probability of an event over time. The specificity of the survival analysis data includes the distribution of events over time and the proportion of classes. Late events are often rare and do not correspond to the main distribution and strongly affect the quality of the models and quality assessment. In this paper, we identify four cases of excessive sensitivity of survival analysis metrics and propose methods to overcome them. To set the equality of observation impacts, we adjust the weights of events based on target time and censoring indicator. According to the sensitivity of metrics, A U P R C (area under Precision-Recall curve) is best suited for assessing the quality of survival models, and other metrics are used as loss functions. To evaluate the influence of the loss function, the B a g g i n g model uses ones to select the size and hyperparameters of the ensemble. The experimental study included eight real medical datasets. The proposed modifications of I B S (Integrated Brier Score) improved the quality of B a g g i n g compared to the classical loss functions. In addition, in seven out of eight datasets, the B a g g i n g with new loss functions outperforms the existing models of the scikit-survival library.

Suggested Citation

  • Iulii Vasilev & Mikhail Petrovskiy & Igor Mashechkin, 2023. "Sensitivity of Survival Analysis Metrics," Mathematics, MDPI, vol. 11(20), pages 1-34, October.
    • Handle: RePEc:gam:jmathe:v:11:y:2023:i:20:p:4246-:d:1257599
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/20/4246/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/11/20/4246/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Uno, Hajime & Cai, Tianxi & Tian, Lu & Wei, L.J., 2007. "Evaluating Prediction Rules for t-Year Survivors With Censored Regression Models," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 527-537, June.
    2. Patrick J. Heagerty & Yingye Zheng, 2005. "Survival Model Predictive Accuracy and ROC Curves," Biometrics, The International Biometric Society, vol. 61(1), pages 92-105, March.
    3. Patrick Royston & Paul C. Lambert, 2011. "Flexible Parametric Survival Analysis Using Stata: Beyond the Cox Model," Stata Press books, StataCorp LP, number fpsaus, March.
    4. Michael Brendel & Arnold Janssen & Claus-Dieter Mayer & Markus Pauly, 2014. "Weighted Logrank Permutation Tests for Randomly Right Censored Life Science Data," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(3), pages 742-761, September.
    5. Lee, Seung-Hwan, 2007. "On the versatility of the combination of the weighted log-rank statistics," Computational Statistics & Data Analysis, Elsevier, vol. 51(12), pages 6557-6564, August.
    6. Håvard Kvamme & Ørnulf Borgan, 2021. "Continuous and discrete-time survival prediction with neural networks," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 27(4), pages 710-736, October.
    7. Pinar Gunel Karadeniz & Ilker Ercan, 2017. "Examining Tests For Comparing Survival Curves With Right Censored Data," Statistics in Transition New Series, Polish Statistical Association, vol. 18(2), pages 311-328, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Matthias Schmid & Thomas Hielscher & Thomas Augustin & Olaf Gefeller, 2011. "A Robust Alternative to the Schemper–Henderson Estimator of Prediction Error," Biometrics, The International Biometric Society, vol. 67(2), pages 524-535, June.
    2. Grzegorz Wyłupek, 2021. "A permutation test for the two-sample right-censored model," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 73(5), pages 1037-1061, October.
    3. Hajime Uno & Tianxi Cai & Lu Tian & L. J. Wei, 2011. "Graphical Procedures for Evaluating Overall and Subject-Specific Incremental Values from New Predictors with Censored Event Time Data," Biometrics, The International Biometric Society, vol. 67(4), pages 1389-1396, December.
    4. Schmid, Matthias & Tutz, Gerhard & Welchowski, Thomas, 2018. "Discrimination measures for discrete time-to-event predictions," Econometrics and Statistics, Elsevier, vol. 7(C), pages 153-164.
    5. Ruosha Li & Jing Ning & Ziding Feng, 2022. "Estimation and inference of predictive discrimination for survival outcome risk prediction models," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 28(2), pages 219-240, April.
    6. Susana Díaz-Coto & Pablo Martínez-Camblor & Sonia Pérez-Fernández, 2020. "smoothROCtime: an R package for time-dependent ROC curve estimation," Computational Statistics, Springer, vol. 35(3), pages 1231-1251, September.
    7. Sehee Kim & Douglas E. Schaubel & Keith P. McCullough, 2018. "A C†index for recurrent event data: Application to hospitalizations among dialysis patients," Biometrics, The International Biometric Society, vol. 74(2), pages 734-743, June.
    8. Jackson, Christopher, 2016. "flexsurv: A Platform for Parametric Survival Modeling in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 70(i08).
    9. Yu Zheng & Tianxi Cai, 2017. "Augmented estimation for t‐year survival with censored regression models," Biometrics, The International Biometric Society, vol. 73(4), pages 1169-1178, December.
    10. Yanyuan Ma & Yuanjia Wang, 2014. "Estimating disease onset distribution functions in mutation carriers with censored mixture data," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 63(1), pages 1-23, January.
    11. Paul Frédéric Blanche & Anders Holt & Thomas Scheike, 2023. "On logistic regression with right censored data, with or without competing risks, and its use for estimating treatment effects," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 29(2), pages 441-482, April.
    12. Pablo Mart�nez-Camblor & Jacobo de U�a-�lvarez & Carmen D�az Corte, 2015. "Expanded renal transplantation: a competing risk model approach," Journal of Applied Statistics, Taylor & Francis Journals, vol. 42(12), pages 2539-2553, December.
    13. Judith H. Parkinson, 2020. "Combined multiple testing of multivariate survival times by censored empirical likelihood," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(3), pages 757-786, September.
    14. Patrick Royston, 2012. "Tools to simulate realistic censored survival-time distributions," Stata Journal, StataCorp LP, vol. 12(4), pages 639-654, December.
    15. Dimitris Rizopoulos, 2011. "Dynamic Predictions and Prospective Accuracy in Joint Models for Longitudinal and Time-to-Event Data," Biometrics, The International Biometric Society, vol. 67(3), pages 819-829, September.
    16. Noori Akhtar-Danesh, 2015. "A Comparison of Modeling Scales in Flexible Parametric Models," 2015 Stata Conference 15, Stata Users Group.
    17. Robin Van Oirbeek & Emmanuel Lesaffre, 2018. "An Investigation of the Discriminatory Ability of the Clustering Effect of the Frailty Survival Model," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 6(3), pages 87-98, April.
    18. Yuanjia Wang & Huaihou Chen & Runze Li & Naihua Duan & Roberto Lewis-Fernández, 2011. "Prediction-Based Structured Variable Selection through the Receiver Operating Characteristic Curves," Biometrics, The International Biometric Society, vol. 67(3), pages 896-905, September.
    19. Enoch Yi-Tung Chen & Yuliya Leontyeva & Chia-Ni Lin & Jung-Der Wang & Mark S. Clements & Paul W. Dickman, 2024. "Comparing Survival Extrapolation within All-Cause and Relative Survival Frameworks by Standard Parametric Models and Flexible Parametric Spline Models Using the Swedish Cancer Registry," Medical Decision Making, , vol. 44(3), pages 269-282, April.
    20. Judit Vitéz-Durgula & Anna Dunay & Gergő Thalmeiner & Balázs Vajai & László Pataki, 2023. "Financial Analysis and Survival Research of the Visegrad Countries’ Health Industries," Sustainability, MDPI, vol. 15(16), pages 1-23, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:20:p:4246-:d:1257599. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.