IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v55y2011i4p1828-1844.html
   My bibliography  Save this article

An experimental comparison of cross-validation techniques for estimating the area under the ROC curve

Author

Listed:
  • Airola, Antti
  • Pahikkala, Tapio
  • Waegeman, Willem
  • De Baets, Bernard
  • Salakoski, Tapio

Abstract

Reliable estimation of the classification performance of inferred predictive models is difficult when working with small data sets. Cross-validation is in this case a typical strategy for estimating the performance. However, many standard approaches to cross-validation suffer from extensive bias or variance when the area under the ROC curve (AUC) is used as the performance measure. This issue is explored through an extensive simulation study. Leave-pair-out cross-validation is proposed for conditional AUC-estimation, as it is almost unbiased, and its deviation variance is as low as that of the best alternative approaches. When using regularized least-squares based learners, efficient algorithms exist for calculating the leave-pair-out cross-validation estimate.

Suggested Citation

  • Airola, Antti & Pahikkala, Tapio & Waegeman, Willem & De Baets, Bernard & Salakoski, Tapio, 2011. "An experimental comparison of cross-validation techniques for estimating the area under the ROC curve," Computational Statistics & Data Analysis, Elsevier, vol. 55(4), pages 1828-1844, April.
    • Handle: RePEc:eee:csdana:v:55:y:2011:i:4:p:1828-1844
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167-9473(10)00446-9
    Download Restriction: Full text for ScienceDirect subscribers only.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rosa A. Schiavo & David J. Hand, 2000. "Ten More Years of Error Rate Research," International Statistical Review, International Statistical Institute, vol. 68(3), pages 295-310, December.
    2. Kim, Ji-Hyun, 2009. "Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap," Computational Statistics & Data Analysis, Elsevier, vol. 53(11), pages 3735-3745, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Campisi, Giovanni & Muzzioli, Silvia & De Baets, Bernard, 2024. "A comparison of machine learning methods for predicting the direction of the US stock market on the basis of volatility indices," International Journal of Forecasting, Elsevier, vol. 40(3), pages 869-880.
    2. Coolen-Maturi, Tahani & Elkhafifi, Faiza F. & Coolen, Frank P.A., 2014. "Three-group ROC analysis: A nonparametric predictive approach," Computational Statistics & Data Analysis, Elsevier, vol. 78(C), pages 69-81.
    3. Riikka Numminen & Ileana Montoya Perez & Ivan Jambor & Tapio Pahikkala & Antti Airola, 2023. "Quicksort leave-pair-out cross-validation for ROC curve analysis," Computational Statistics, Springer, vol. 38(3), pages 1579-1595, September.
    4. Zatonatska Tetiana & Dluhopolskyi Oleksandr & Artyukh Tatiana & Tymchenko Kateryna, 2022. "Forecasting the Behavior of Target Segments to Activate Advertising Tools: Case of Mobile Operator Vodafone Ukraine," Economics, Sciendo, vol. 10(1), pages 87-104, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Conde David & Salvador Bonifacio & Rueda Cristina & Fernández Miguel A., 2013. "Performance and estimation of the true error rate of classification rules built with additional information. An application to a cancer trial," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 12(5), pages 583-602, October.
    2. Ha, Tran Vinh & Asada, Takumi & Arimura, Mikiharu, 2019. "Determination of the influence factors on household vehicle ownership patterns in Phnom Penh using statistical and machine learning methods," Journal of Transport Geography, Elsevier, vol. 78(C), pages 70-86.
    3. Mark G E White & Neil E Bezodis & Jonathon Neville & Huw Summers & Paul Rees, 2022. "Determining jumping performance from a single body-worn accelerometer using machine learning," PLOS ONE, Public Library of Science, vol. 17(2), pages 1-25, February.
    4. Erfan Mahmoodi & Mahmood Azari & Mohammad Taghi Dastorani & Aryan Salvati, 2024. "Comparison of Hydrological Modeling, Artificial Neural Networks and Multi-Criteria Decision Making Approaches for Determining Flood Source Areas," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 38(13), pages 5343-5363, October.
    5. Charlotte Soneson & Sarah Gerster & Mauro Delorenzi, 2014. "Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation," PLOS ONE, Public Library of Science, vol. 9(6), pages 1-13, June.
    6. Richard A. Johansen & Molly K. Reif & Christina L. Saltus & Kaytee L. Pokrzywinski, 2024. "A Broadscale Assessment of Sentinel-2 Imagery and the Google Earth Engine for the Nationwide Mapping of Chlorophyll a," Sustainability, MDPI, vol. 16(5), pages 1-17, March.
    7. Dean Fantazzini & Yufeng Xiao, 2023. "Detecting Pump-and-Dumps with Crypto-Assets: Dealing with Imbalanced Datasets and Insiders’ Anticipated Purchases," Econometrics, MDPI, vol. 11(3), pages 1-73, August.
    8. Zhengnan Huang & Hongjiu Zhang & Jonathan Boss & Stephen A Goutman & Bhramar Mukherjee & Ivo D Dinov & Yuanfang Guan & for the Pooled Resource Open-Access ALS Clinical Trials Consortium, 2017. "Complete hazard ranking to analyze right-censored data: An ALS survival study," PLOS Computational Biology, Public Library of Science, vol. 13(12), pages 1-21, December.
    9. Xue, Jing-Hao & Titterington, D. Michael, 2010. "On the generative-discriminative tradeoff approach: Interpretation, asymptotic efficiency and classification performance," Computational Statistics & Data Analysis, Elsevier, vol. 54(2), pages 438-451, February.
    10. Vinícius Serafini Roglio & Eduardo Nunes Borges & Francisco Diego Rabelo-da-Ponte & Felipe Ornell & Juliana Nichterwitz Scherer & Jaqueline Bohrer Schuch & Ives Cavalcante Passos & Breno Sanvicente-Vi, 2020. "Prediction of attempted suicide in men and women with crack-cocaine use disorder in Brazil," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-19, May.
    11. Gianluca Gazzola & Myong K. Jeong, 2021. "Support vector regression for polyhedral and missing data," Annals of Operations Research, Springer, vol. 303(1), pages 483-506, August.
    12. Ayed Alwadain & Rao Faizan Ali & Amgad Muneer, 2023. "Estimating Financial Fraud through Transaction-Level Features and Machine Learning," Mathematics, MDPI, vol. 11(5), pages 1-15, February.
    13. John J. Nay & Yevgeniy Vorobeychik, 2016. "Predicting Human Cooperation," Papers 1601.07792, arXiv.org, revised Apr 2016.
    14. Matthias Schmid & Thomas Hielscher & Thomas Augustin & Olaf Gefeller, 2011. "A Robust Alternative to the Schemper–Henderson Estimator of Prediction Error," Biometrics, The International Biometric Society, vol. 67(2), pages 524-535, June.
    15. Shusaku Tsumoto & Tomohiro Kimura & Shoji Hirano, 2022. "Expectation–Maximization (EM) Clustering as a Preprocessing Method for Clinical Pathway Mining," The Review of Socionetwork Strategies, Springer, vol. 16(1), pages 25-52, April.
    16. Luts, Jan & Ormerod, John T., 2014. "Mean field variational Bayesian inference for support vector machine classification," Computational Statistics & Data Analysis, Elsevier, vol. 73(C), pages 163-176.
    17. David Rios Insua & Roi Naveiro & Victor Gallego, 2020. "Perspectives on Adversarial Classification," Mathematics, MDPI, vol. 8(11), pages 1-21, November.
    18. Dietrichson, Jens & Klokker, Rasmus H., 2024. "Predicting preschool problems," Children and Youth Services Review, Elsevier, vol. 161(C).
    19. John J Nay & Yevgeniy Vorobeychik, 2016. "Predicting Human Cooperation," PLOS ONE, Public Library of Science, vol. 11(5), pages 1-19, May.
    20. Kazim Topuz & Behrooz Davazdahemami & Dursun Delen, 2024. "A Bayesian belief network-based analytics methodology for early-stage risk detection of novel diseases," Annals of Operations Research, Springer, vol. 341(1), pages 673-697, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:55:y:2011:i:4:p:1828-1844. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.