IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i22p4206-d968911.html
   My bibliography  Save this article

Convergence Behavior of Optimal Cut-Off Points Derived from Receiver Operating Characteristics Curve Analysis: A Simulation Study

Author

Listed:
  • Oke Gerke

    (Department of Nuclear Medicine, Odense University Hospital, 5000 Odense, Denmark
    Department of Clinical Research, University of Southern Denmark, 5000 Odense, Denmark)

  • Antonia Zapf

    (Department of Medical Biometry and Epidemiology, University Medical Center Hamburg-Eppendorf, 20251 Hamburg, Germany)

Abstract

The area under the receiver operating characteristics curve is a popular measure of the overall discriminatory power of a continuous variable used to indicate the presence of an outcome of interest, such as disease or disease progression. In clinical practice, the use of cut-off points as benchmark values for further treatment planning is greatly appreciated, despite the loss of information that such a dichotomization implies. Optimal cut-off points are often derived from fixed sample size studies, and the aim of this study was to investigate the convergence behavior of optimal cut-off points with increasing sample size and to explore a heuristic and path-based algorithm for cut-off point determination that targets stagnating cut-off point values. To this end, the closest-to-(0,1) criterion in receiver operating characteristics curve analysis was used, and the heuristic and path-based algorithm aimed at cut-off points that deviated less than 1% from the cut-off point of the previous iteration. Such a heuristic determination stopped after only a few iterations, thereby implicating practicable sample sizes; however, the result was, at best, a rough estimate of an optimal cut-off point that was unbiased and positively and negatively biased for a prevalence of 0.5, smaller than 0.5, and larger than 0.5, respectively.

Suggested Citation

  • Oke Gerke & Antonia Zapf, 2022. "Convergence Behavior of Optimal Cut-Off Points Derived from Receiver Operating Characteristics Curve Analysis: A Simulation Study," Mathematics, MDPI, vol. 10(22), pages 1-14, November.
  • Handle: RePEc:gam:jmathe:v:10:y:2022:i:22:p:4206-:d:968911
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/22/4206/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2227-7390/10/22/4206/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Margaret S. Pepe & Gary Longton & Holly Janes, 2009. "Estimation and comparison of receiver operating characteristic curves," Stata Journal, StataCorp LP, vol. 9(1), pages 1-16, March.
    2. William M. Briggs & Russell Zaretzki, 2008. "The Skill Plot: A Graphical Technique for Evaluating Continuous Diagnostic Tests," Biometrics, The International Biometric Society, vol. 64(1), pages 250-256, March.
    3. Andra Ciocan & Nadim Al Hajjar & Florin Graur & Valentin C. Oprea & Răzvan A. Ciocan & Sorana D. Bolboacă, 2020. "Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example," Mathematics, MDPI, vol. 8(10), pages 1-17, October.
    4. Karim O. Hajian-Tilaki & James A. Hanley & Lawrence Joseph & Jean-Paul Collet, 1997. "A Comparison of Parametric and Nonparametric Approaches to ROC Analysis of Quantitative Diagnostic Tests," Medical Decision Making, , vol. 17(1), pages 94-102, February.
    5. George Laking & Joanne Lord & Alastair Fischer, 2006. "The economics of diagnosis," Health Economics, John Wiley & Sons, Ltd., vol. 15(10), pages 1109-1120, October.
    6. López-Ratón, Mónica & Rodríguez-Álvarez, María Xosé & Cadarso-Suárez, Carmen & Gude-Sampedro, Francisco, 2014. "OptimalCutpoints: An R Package for Selecting Optimal Cutpoints in Diagnostic Tests," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 61(i08).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bespalova, Olga, 2018. "Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA," MPRA Paper 117706, University Library of Munich, Germany.
    2. Marcin Łupiński, 2019. "Wskaźniki wczesnego ostrzegania przed niestabilnością finansową polskiego sektora bankowego," Collegium of Economic Analysis Annals, Warsaw School of Economics, Collegium of Economic Analysis, issue 55, pages 99-113.
    3. Yuanjia Wang & Huaihou Chen & Runze Li & Naihua Duan & Roberto Lewis-Fernández, 2011. "Prediction-Based Structured Variable Selection through the Receiver Operating Characteristic Curves," Biometrics, The International Biometric Society, vol. 67(3), pages 896-905, September.
    4. Kinga Polanska & Anna Krol & Pawel Kaluzny & Danuta Ligocka & Karolina Mikolajewska & Seif Shaheen & Robert Walton & Wojciech Hanke, 2016. "Estimation of Saliva Cotinine Cut-Off Points for Active and Passive Smoking during Pregnancy—Polish Mother and Child Cohort (REPRO_PL)," IJERPH, MDPI, vol. 13(12), pages 1-15, December.
    5. Holly Janes & Gary Longton & Margaret S. Pepe, 2009. "Accommodating covariates in receiver operating characteristic analysis," Stata Journal, StataCorp LP, vol. 9(1), pages 17-39, March.
    6. Gregory S. Zaric, 2008. "Optimal drug pricing, limited use conditions and stratified net benefits for Markov models of disease progression," Health Economics, John Wiley & Sons, Ltd., vol. 17(11), pages 1277-1294, November.
    7. André K. Anundsen & Karsten Gerdrup & Frank Hansen & Kasper Kragh‐Sørensen, 2016. "Bubbles and Crises: The Role of House Prices and Credit," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 31(7), pages 1291-1311, November.
    8. Sandra Viz-Lasheras & Alberto Gómez-Carballa & Xabier Bello & Irene Rivero-Calle & Ana Isabel Dacosta & Myrsini Kaforou & Dominic Habgood-Coote & Aubrey J. Cunnington & Marieke Emonts & Jethro A. Herb, 2025. "A diagnostic host-specific transcriptome response for Mycoplasma pneumoniae pneumonia to guide pediatric patient treatment," Nature Communications, Nature, vol. 16(1), pages 1-13, December.
    9. John Muschelli, 2020. "ROC and AUC with a Binary Predictor: a Potentially Misleading Metric," Journal of Classification, Springer;The Classification Society, vol. 37(3), pages 696-708, October.
    10. Drehmann, Mathias & Juselius, Mikael, 2014. "Evaluating early warning indicators of banking crises: Satisfying policy requirements," International Journal of Forecasting, Elsevier, vol. 30(3), pages 759-780.
    11. Baker Stuart G. & Van Calster Ben & Steyerberg Ewout W., 2012. "Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update," The International Journal of Biostatistics, De Gruyter, vol. 8(1), pages 1-37, March.
    12. David Bardey & Philippe de Donder & Vera Zaporozhets, 2024. "The Health Technology Assessment Approach of the Economic Value of Diagnostic Tests - A Literature Review," Working Papers hal-04472485, HAL.
    13. K. Drakopoulos & R. S. Randhawa, 2021. "Why Perfect Tests May Not Be Worth Waiting For: Information as a Commodity," Management Science, INFORMS, vol. 67(11), pages 6678-6693, November.
    14. Yusuf Yıldırım & Anirban Sanyal, 2022. "Evaluating the Effectiveness of Early Warning Indicators: An Application of Receiver Operating Characteristic Curve Approach to Panel Data," Scientific Annals of Economics and Business (continues Analele Stiintifice), Alexandru Ioan Cuza University, Faculty of Economics and Business Administration, vol. 69(4), pages 557-597, December.
    15. Rocío Aznar-Gimeno & Luis M. Esteban & Gorka Labata-Lezaun & Rafael del-Hoyo-Alonso & David Abadia-Gallego & J. Ramón Paño-Pardo & M. José Esquillor-Rodrigo & Ángel Lanas & M. Trinidad Serrano, 2021. "A Clinical Decision Web to Predict ICU Admission or Death for Patients Hospitalised with COVID-19 Using Machine Learning Algorithms," IJERPH, MDPI, vol. 18(16), pages 1-20, August.
    16. Jesús F. Lampón & Pablo Cabanelas-Lorenzo & Santiago Lago-Peñas, 2013. "Why firms relocate their production overseas? The answer lies inside: corporate, logistic and technological determinants," Working Papers 2013/3, Institut d'Economia de Barcelona (IEB).
    17. Dean Fantazzini & Yufeng Xiao, 2023. "Detecting Pump-and-Dumps with Crypto-Assets: Dealing with Imbalanced Datasets and Insiders’ Anticipated Purchases," Econometrics, MDPI, vol. 11(3), pages 1-73, August.
    18. Jesús F. Lampón & Pablo Cabanelas-Lorenzo & Santiago Lago-Peñas, 2013. "Why firms relocate their production overseas? The answer lies inside: corporate, logistic and technological determinants," Working Papers 2013/3, Institut d'Economia de Barcelona (IEB).
    19. Elizabeth Gutierrez & Jake Krupa & Miguel Minutti-Meza & Maria Vulcheva, 2020. "Do going concern opinions provide incremental information to predict corporate defaults?," Review of Accounting Studies, Springer, vol. 25(4), pages 1344-1381, December.
    20. Xinyue Wu & Hong Zhu & Liuru Hu & Jian Meng & Fulu Sun, 2024. "Analysis of Short-Term Heavy Rainfall-Based Urban Flood Disaster Risk Assessment Using Integrated Learning Approach," Sustainability, MDPI, vol. 16(18), pages 1-19, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:22:p:4206-:d:968911. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.