IDEAS home Printed from https://ideas.repec.org/a/bla/biomet/v78y2022i1p352-363.html
   My bibliography  Save this article

A closed max‐t test for multiple comparisons of areas under the ROC curve

Author

Listed:
  • Paul Blanche
  • Jean‐François Dartigues
  • Jérémie Riou

Abstract

Comparing areas under the ROC curve (AUCs) is a popular approach to compare prognostic biomarkers. The aim of this paper is to present an efficient method to control the family‐wise error rate when multiple comparisons are performed. We suggest to combine the max‐t test and the closed testing procedures. We build on previous work on asymptotic results for ROC curves and on general multiple testing methods to efficiently take into account both the correlations between the test statistics and the logical constraints between the null hypotheses. The proposed method results in an uniformly more powerful procedure than both the single‐step max‐t test procedure and popular stepwise extensions of the Bonferroni procedure, such as Bonferroni–Holm. As demonstrated in this paper, the method can be applied in most usual contexts, including the time‐dependent context with right censored data. We show how the method works in practice through a motivating example where we compare several psychometric scores to predict the t‐year risk of Alzheimer's disease. The example illustrates several multiple testing settings and demonstrates the advantage of using the proposed methods over common alternatives. R code has been made available to facilitate the use of the methods by others.

Suggested Citation

  • Paul Blanche & Jean‐François Dartigues & Jérémie Riou, 2022. "A closed max‐t test for multiple comparisons of areas under the ROC curve," Biometrics, The International Biometric Society, vol. 78(1), pages 352-363, March.
    • Handle: RePEc:bla:biomet:v:78:y:2022:i:1:p:352-363
    DOI: 10.1111/biom.13401
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/biom.13401
    Download Restriction: no
    File URL: https://libkey.io/10.1111/biom.13401?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Christian Bressen Pipper & Christian Ritz & Hans Bisgaard, 2012. "A versatile method for confirmatory evaluation of the effects of a covariate in multiple models," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 61(2), pages 315-326, March.
    2. Uno, Hajime & Cai, Tianxi & Tian, Lu & Wei, L.J., 2007. "Evaluating Prediction Rules for t-Year Survivors With Censored Regression Models," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 527-537, June.
    3. Westfall, Peter H. & Tobias, Randall D., 2007. "Multiple Testing of General Contrasts: Truncated Closure and the Extended ShafferRoyen Method," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 487-494, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brice Ozenne & Esben Budtz-Jørgensen & Sebastian Elgaard Ebert, 2023. "Controlling the familywise error rate when performing multiple comparisons in a linear latent variable model," Computational Statistics, Springer, vol. 38(1), pages 1-23, March.
    2. Schmid, Matthias & Tutz, Gerhard & Welchowski, Thomas, 2018. "Discrimination measures for discrete time-to-event predictions," Econometrics and Statistics, Elsevier, vol. 7(C), pages 153-164.
    3. Yu Zheng & Tianxi Cai, 2017. "Augmented estimation for t‐year survival with censored regression models," Biometrics, The International Biometric Society, vol. 73(4), pages 1169-1178, December.
    4. Alessandro, Martin & Cardinale Lagomarsino, Bruno & Scartascini, Carlos & Streb, Jorge & Torrealday, Jerónimo, 2021. "Transparency and Trust in Government. Evidence from a Survey Experiment," World Development, Elsevier, vol. 138(C).
    5. Holger Dette & Laura Hoyden & Sonja Kuhnt & Kirsten Schorning, 2017. "Optimal designs for thermal spraying," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 66(1), pages 53-72, January.
    6. Paul Frédéric Blanche & Anders Holt & Thomas Scheike, 2023. "On logistic regression with right censored data, with or without competing risks, and its use for estimating treatment effects," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 29(2), pages 441-482, April.
    7. Patrizia Pochetti & Danila Azzolina & Beatrice Ragnoli & Paolo Amedeo Tillio & Vincenzo Cantaluppi & Mario Malerba, 2020. "Interrelationship among Obstructive Sleep Apnea, Renal Function and Survival: A Cohort Study," IJERPH, MDPI, vol. 17(14), pages 1-12, July.
    8. Frank Schaarschmidt & Christian Ritz & Ludwig A. Hothorn, 2022. "The Tukey trend test: Multiplicity adjustment using multiple marginal models," Biometrics, The International Biometric Society, vol. 78(2), pages 789-797, June.
    9. Ningning Xu & Aldo Solari & Jelle J. Goeman, 2023. "Closed testing with Globaltest, with application in metabolomics," Biometrics, The International Biometric Society, vol. 79(2), pages 1103-1113, June.
    10. Ruosha Li & Jing Ning & Ziding Feng, 2022. "Estimation and inference of predictive discrimination for survival outcome risk prediction models," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 28(2), pages 219-240, April.
    11. Signe M. Jensen & Christian Ritz, 2015. "Simultaneous Inference for Model Averaging of Derived Parameters," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 68-76, January.
    12. He Kevin & Zhou Xiang & Jiang Hui & Wen Xiaoquan & Li Yi, 2018. "False discovery control for penalized variable selections with high-dimensional covariates," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 17(6), pages 1-11, December.
    13. Susana Díaz-Coto & Pablo Martínez-Camblor & Sonia Pérez-Fernández, 2020. "smoothROCtime: an R package for time-dependent ROC curve estimation," Computational Statistics, Springer, vol. 35(3), pages 1231-1251, September.
    14. Matthias Schmid & Thomas Hielscher & Thomas Augustin & Olaf Gefeller, 2011. "A Robust Alternative to the Schemper–Henderson Estimator of Prediction Error," Biometrics, The International Biometric Society, vol. 67(2), pages 524-535, June.
    15. Marian Reiff & Erik Šoltés & Silvia Komara & Tatiana Šoltésová & Silvia Zelinová, 2022. "Segmentation and estimation of claim severity in motor third-party liability insurance through contrast analysis," Equilibrium. Quarterly Journal of Economics and Economic Policy, Institute of Economic Research, vol. 17(3), pages 803-842, September.
    16. Cheng Zheng & Yingye Zheng, 2019. "Calibrating Variations in Biomarker Measures for Improving Prediction with Time-to-event Outcomes," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 11(3), pages 477-503, December.
    17. Denis Agniel & Katherine P. Liao & Tianxi Cai, 2016. "Estimation and testing for multiple regulation of multivariate mixed outcomes," Biometrics, The International Biometric Society, vol. 72(4), pages 1194-1205, December.
    18. Mingzhu Liu & Chirag Nagpal & Artur Dubrawski, 2024. "Deep Survival Models Can Improve Long-Term Mortality Risk Estimates from Chest Radiographs," Forecasting, MDPI, vol. 6(2), pages 1-14, May.
    19. Glenn Heller, 2021. "The added value of new covariates to the brier score in cox survival models," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 27(1), pages 1-14, January.
    20. Dendramis, Y. & Tzavalis, E. & Varthalitis, P. & Athanasiou, E., 2020. "Predicting default risk under asymmetric binary link functions," International Journal of Forecasting, Elsevier, vol. 36(3), pages 1039-1056.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:biomet:v:78:y:2022:i:1:p:352-363. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0006-341X .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.