IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i20p4246-d1257599.html
   My bibliography  Save this article

Sensitivity of Survival Analysis Metrics

Author

Listed:
  • Iulii Vasilev

    (Computer Science Department, Lomonosov Moscow State University, Vorobjovy Gory, 119899 Moscow, Russia)

  • Mikhail Petrovskiy

    (Computer Science Department, Lomonosov Moscow State University, Vorobjovy Gory, 119899 Moscow, Russia)

  • Igor Mashechkin

    (Computer Science Department, Lomonosov Moscow State University, Vorobjovy Gory, 119899 Moscow, Russia)

Abstract

Survival analysis models allow for predicting the probability of an event over time. The specificity of the survival analysis data includes the distribution of events over time and the proportion of classes. Late events are often rare and do not correspond to the main distribution and strongly affect the quality of the models and quality assessment. In this paper, we identify four cases of excessive sensitivity of survival analysis metrics and propose methods to overcome them. To set the equality of observation impacts, we adjust the weights of events based on target time and censoring indicator. According to the sensitivity of metrics, A U P R C (area under Precision-Recall curve) is best suited for assessing the quality of survival models, and other metrics are used as loss functions. To evaluate the influence of the loss function, the B a g g i n g model uses ones to select the size and hyperparameters of the ensemble. The experimental study included eight real medical datasets. The proposed modifications of I B S (Integrated Brier Score) improved the quality of B a g g i n g compared to the classical loss functions. In addition, in seven out of eight datasets, the B a g g i n g with new loss functions outperforms the existing models of the scikit-survival library.

Suggested Citation

  • Iulii Vasilev & Mikhail Petrovskiy & Igor Mashechkin, 2023. "Sensitivity of Survival Analysis Metrics," Mathematics, MDPI, vol. 11(20), pages 1-34, October.
    • Handle: RePEc:gam:jmathe:v:11:y:2023:i:20:p:4246-:d:1257599
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/20/4246/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/11/20/4246/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Patrick Royston & Paul C. Lambert, 2011. "Flexible Parametric Survival Analysis Using Stata: Beyond the Cox Model," Stata Press books, StataCorp LLC, number fpsaus, August.
    2. Lee, Seung-Hwan, 2007. "On the versatility of the combination of the weighted log-rank statistics," Computational Statistics & Data Analysis, Elsevier, vol. 51(12), pages 6557-6564, August.
    3. Pinar Gunel Karadeniz & Ilker Ercan, 2017. "Examining Tests For Comparing Survival Curves With Right Censored Data," Statistics in Transition New Series, Polish Statistical Association, vol. 18(2), pages 311-328, June.
    4. Håvard Kvamme & Ørnulf Borgan, 2021. "Continuous and discrete-time survival prediction with neural networks," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 27(4), pages 710-736, October.
    5. Uno, Hajime & Cai, Tianxi & Tian, Lu & Wei, L.J., 2007. "Evaluating Prediction Rules for t-Year Survivors With Censored Regression Models," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 527-537, June.
    6. Michael Brendel & Arnold Janssen & Claus-Dieter Mayer & Markus Pauly, 2014. "Weighted Logrank Permutation Tests for Randomly Right Censored Life Science Data," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(3), pages 742-761, September.
    7. Patrick J. Heagerty & Yingye Zheng, 2005. "Survival Model Predictive Accuracy and ROC Curves," Biometrics, The International Biometric Society, vol. 61(1), pages 92-105, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ruosha Li & Jing Ning & Ziding Feng, 2022. "Estimation and inference of predictive discrimination for survival outcome risk prediction models," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 28(2), pages 219-240, April.
    2. Susana Díaz-Coto & Pablo Martínez-Camblor & Sonia Pérez-Fernández, 2020. "smoothROCtime: an R package for time-dependent ROC curve estimation," Computational Statistics, Springer, vol. 35(3), pages 1231-1251, September.
    3. Sehee Kim & Douglas E. Schaubel & Keith P. McCullough, 2018. "A C†index for recurrent event data: Application to hospitalizations among dialysis patients," Biometrics, The International Biometric Society, vol. 74(2), pages 734-743, June.
    4. Matthias Schmid & Thomas Hielscher & Thomas Augustin & Olaf Gefeller, 2011. "A Robust Alternative to the Schemper–Henderson Estimator of Prediction Error," Biometrics, The International Biometric Society, vol. 67(2), pages 524-535, June.
    5. Grzegorz Wyłupek, 2021. "A permutation test for the two-sample right-censored model," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 73(5), pages 1037-1061, October.
    6. Hajime Uno & Tianxi Cai & Lu Tian & L. J. Wei, 2011. "Graphical Procedures for Evaluating Overall and Subject-Specific Incremental Values from New Predictors with Censored Event Time Data," Biometrics, The International Biometric Society, vol. 67(4), pages 1389-1396, December.
    7. Schmid, Matthias & Tutz, Gerhard & Welchowski, Thomas, 2018. "Discrimination measures for discrete time-to-event predictions," Econometrics and Statistics, Elsevier, vol. 7(C), pages 153-164.
    8. Yu Zheng & Tianxi Cai, 2017. "Augmented estimation for t‐year survival with censored regression models," Biometrics, The International Biometric Society, vol. 73(4), pages 1169-1178, December.
    9. Paul Frédéric Blanche & Anders Holt & Thomas Scheike, 2023. "On logistic regression with right censored data, with or without competing risks, and its use for estimating treatment effects," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 29(2), pages 441-482, April.
    10. Judith H. Parkinson, 2020. "Combined multiple testing of multivariate survival times by censored empirical likelihood," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(3), pages 757-786, September.
    11. Robin Van Oirbeek & Emmanuel Lesaffre, 2018. "An Investigation of the Discriminatory Ability of the Clustering Effect of the Frailty Survival Model," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 6(3), pages 87-98, April.
    12. Yuanjia Wang & Huaihou Chen & Runze Li & Naihua Duan & Roberto Lewis-Fernández, 2011. "Prediction-Based Structured Variable Selection through the Receiver Operating Characteristic Curves," Biometrics, The International Biometric Society, vol. 67(3), pages 896-905, September.
    13. Kevin He & Yue Wang & Xiang Zhou & Han Xu & Can Huang, 2019. "An improved variable selection procedure for adaptive Lasso in high-dimensional survival analysis," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 25(3), pages 569-585, July.
    14. Weining Shen & Jing Ning & Ying Yuan, 2015. "A direct method to evaluate the time-dependent predictive accuracy for biomarkers," Biometrics, The International Biometric Society, vol. 71(2), pages 439-449, June.
    15. Stefano Bonnini & Getnet Melak Assegie & Kamila Trzcinska, 2024. "Review about the Permutation Approach in Hypothesis Testing," Mathematics, MDPI, vol. 12(17), pages 1-29, August.
    16. Iversen, Tor & Ching-to , Albert Ma, 2020. "Technology Adoption in Primary Health Care," HERO Online Working Paper Series 2020:4, University of Oslo, Health Economics Research Programme.
    17. Shanshan Li & Yang Ning, 2015. "Estimation of covariate‐specific time‐dependent ROC curves in the presence of missing biomarkers," Biometrics, The International Biometric Society, vol. 71(3), pages 666-676, September.
    18. Herrera Dappe,Matias & Melecky,Martin & Turkgulu,Burak, 2022. "Fiscal Risks from Early Termination of Public-Private Partnerships in Infrastructure," Policy Research Working Paper Series 9972, The World Bank.
    19. Michael J. Crowther & Paul C. Lambert, 2012. "Simulating complex survival data," Stata Journal, StataCorp LP, vol. 12(4), pages 674-687, December.
    20. Dennis Dobler & Markus Pauly, 2018. "Bootstrap- and permutation-based inference for the Mann–Whitney effect for right-censored and tied data," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 27(3), pages 639-658, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:20:p:4246-:d:1257599. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.