IDEAS home Printed from https://ideas.repec.org/a/gam/jstats/v6y2023i2p34-538d1131818.html
   My bibliography  Save this article

Evaluation of Risk Prediction with Hierarchical Data: Dependency Adjusted Confidence Intervals for the AUC

Author

Listed:
  • Camden Bay

    (Brigham and Women’s Hospital, Harvard Medical School, Boston, MA 02115, USA)

  • Robert J Glynn

    (Brigham and Women’s Hospital, Harvard Medical School, Boston, MA 02115, USA)

  • Johanna M Seddon

    (University of Massachusetts Chan Medical School, Worcester, MA 01655, USA)

  • Mei-Ling Ting Lee

    (Department of Epidemiology and Biostatistics, University of Maryland School of Public Health, College Park, MD 20742, USA)

  • Bernard Rosner

    (Brigham and Women’s Hospital, Harvard Medical School, Boston, MA 02115, USA)

Abstract

The area under the true ROC curve (AUC) is routinely used to determine how strongly a given model discriminates between the levels of a binary outcome. Standard inference with the AUC requires that outcomes be independent of each other. To overcome this limitation, a method was developed for the estimation of the variance of the AUC in the setting of two-level hierarchical data using probit-transformed prediction scores generated from generalized estimating equation models, thereby allowing for the application of inferential methods. This manuscript presents an extension of this approach so that inference for the AUC may be performed in a three-level hierarchical data setting (e.g., eyes nested within persons and persons nested within families). A method that accounts for the effect of tied prediction scores on inference is also described. The performance of 95% confidence intervals around the AUC was assessed through the simulation of three-level clustered data in multiple settings, including ones with tied data and variable cluster sizes. Across all settings, the actual 95% confidence interval coverage varied from 0.943 to 0.958, and the ratio of the theoretical variance to the empirical variance of the AUC varied from 0.920 to 1.013. The results are better than those from existing methods. Two examples of applying the proposed methodology are presented.

Suggested Citation

  • Camden Bay & Robert J Glynn & Johanna M Seddon & Mei-Ling Ting Lee & Bernard Rosner, 2023. "Evaluation of Risk Prediction with Hierarchical Data: Dependency Adjusted Confidence Intervals for the AUC," Stats, MDPI, vol. 6(2), pages 1-13, April.
  • Handle: RePEc:gam:jstats:v:6:y:2023:i:2:p:34-538:d:1131818
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2571-905X/6/2/34/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2571-905X/6/2/34/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bernard Rosner & Robert J. Glynn & Mei-Ling T. Lee, 2006. "Extension of the Rank Sum Test for Clustered Data: Two-Group Comparisons with Group Membership Defined at the Subunit Level," Biometrics, The International Biometric Society, vol. 62(4), pages 1251-1259, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Fu, Liya & Wang, You-Gan & Bai, Zhidong, 2010. "Rank regression for analysis of clustered data: A natural induced smoothing approach," Computational Statistics & Data Analysis, Elsevier, vol. 54(4), pages 1036-1050, April.
    2. Haataja, Riina & Larocque, Denis & Nevalainen, Jaakko & Oja, Hannu, 2009. "A weighted multivariate signed-rank test for cluster-correlated data," Journal of Multivariate Analysis, Elsevier, vol. 100(6), pages 1107-1119, July.
    3. Atsushi Kawaguchi & Gary G. Koch & Ratna Ramaswamy, 2009. "Applications of Extensions of Bivariate Rank Sum Statistics to the Crossover Design to Compare Two Treatments Through Four Sequence Groups," Biometrics, The International Biometric Society, vol. 65(3), pages 979-988, September.
    4. Denis Larocque & Riina Haataja & Jaakko Nevalainen & Hannu Oja, 2010. "Two sample tests for the nonparametric Behrens–Fisher problem with clustered data," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 22(6), pages 755-771.
    5. Riina Lemponen & Denis Larocque & Jaakko Nevalainen & Hannu Oja, 2012. "Weighted rank tests and Hodges-Lehmann estimates for the multivariate two-sample location problem with clustered data," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 24(4), pages 977-991, December.
    6. Lan Wang & Runze Li, 2009. "Weighted Wilcoxon-Type Smoothly Clipped Absolute Deviation Method," Biometrics, The International Biometric Society, vol. 65(2), pages 564-571, June.
    7. Bernard Rosner & Robert J. Glynn, 2011. "Power and Sample Size Estimation for the Clustered Wilcoxon Test," Biometrics, The International Biometric Society, vol. 67(2), pages 646-653, June.
    8. Konietschke, F. & Harrar, S.W. & Lange, K. & Brunner, E., 2012. "Ranking procedures for matched pairs with missing data — Asymptotic theory and a small sample approximation," Computational Statistics & Data Analysis, Elsevier, vol. 56(5), pages 1090-1102.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jstats:v:6:y:2023:i:2:p:34-538:d:1131818. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.