IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v8y2020i10p1741-d425988.html
   My bibliography  Save this article

Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example

Author

Listed:
  • Andra Ciocan

    (Department of Medical Informatics and Biostatistics, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Louis Pasteur Street, No. 6, 400349 Cluj-Napoca, Romania
    “Prof. Dr. Octavian Fodor” Regional Institute of Gastroenterology and Hepatology Cluj-Napoca, Croitorilor Street, No. 19–21, 400162 Cluj-Napoca, Romania)

  • Nadim Al Hajjar

    (“Prof. Dr. Octavian Fodor” Regional Institute of Gastroenterology and Hepatology Cluj-Napoca, Croitorilor Street, No. 19–21, 400162 Cluj-Napoca, Romania
    Department of Surgery, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Croitorilor Street, No. 19–21, 400162 Cluj-Napoca, Romania)

  • Florin Graur

    (“Prof. Dr. Octavian Fodor” Regional Institute of Gastroenterology and Hepatology Cluj-Napoca, Croitorilor Street, No. 19–21, 400162 Cluj-Napoca, Romania
    Department of Surgery, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Croitorilor Street, No. 19–21, 400162 Cluj-Napoca, Romania)

  • Valentin C. Oprea

    (“Dr. Constantin Papilian” Military Emergency Hospital Cluj-Napoca, General Traian Moșoiu Street, No. 22, 400132 Cluj-Napoca, Romania)

  • Răzvan A. Ciocan

    (Department of Medical Skills—Human Sciences, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Marinescu Street, No. 23, 400337 Cluj-Napoca, Romania)

  • Sorana D. Bolboacă

    (Department of Medical Informatics and Biostatistics, “Iuliu Hațieganu” University of Medicine and Pharmacy Cluj-Napoca, Louis Pasteur Street, No. 6, 400349 Cluj-Napoca, Romania)

Abstract

The stability of receiver operating characteristic in context of random split used in development and validation sets, as compared to the full models for three inflammatory ratios (neutrophil-to-lymphocyte (NLR), derived neutrophil-to-lymphocyte (dNLR) and platelet-to-lymphocyte (PLR) ratio) evaluated as predictors for metastasis in patients with colorectal cancer, was investigated. Data belonging to patients admitted with the diagnosis of colorectal cancer from January 2014 until September 2019 in a single hospital were used. There were 1688 patients eligible for the study, 418 in the metastatic stage. All investigated inflammatory ratios proved to be significant classification models on both the full models and on cross-validations (AUCs > 0.05). High variability of the cut-off values was observed in the unrestricted and restricted split (full models: 4.255 for NLR, 2.745 for dNLR and 255.56 for PLR; random splits: cut-off from 3.215 to 5.905 for NLR, from 2.625 to 3.575 for dNLR and from 134.67 to 335.9 for PLR), but with no effect on the models characteristics or performances. The investigated biomarkes proved limited value as predictors for metastasis (AUCs < 0.8), with largely sensitivity and specificity (from 33.3% to 79.2% for the full model and 29.1% to 82.7% in the restricted splits). Our results showed that a simple random split of observations, weighting or not the patients with and whithout metastasis, in a ROC analysis assures the performances similar to the full model, if at least 70% of the available population is included in the study.

Suggested Citation

  • Andra Ciocan & Nadim Al Hajjar & Florin Graur & Valentin C. Oprea & Răzvan A. Ciocan & Sorana D. Bolboacă, 2020. "Receiver Operating Characteristic Prediction for Classification: Performances in Cross-Validation by Example," Mathematics, MDPI, vol. 8(10), pages 1-17, October.
    • Handle: RePEc:gam:jmathe:v:8:y:2020:i:10:p:1741-:d:425988
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/8/10/1741/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/8/10/1741/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Charles E. Phelps & Alan Hutson, 1995. "Estimating Diagnostic Test Accuracy Using a "Fuzzy Gold Standard"," Medical Decision Making, , vol. 15(1), pages 44-57, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Aleksandr Kulikov & Anton Loskutov & Dmitriy Bezdushniy & Ilya Petrov, 2023. "Decision Tree Models and Machine Learning Algorithms in the Fault Recognition on Power Lines with Branches," Energies, MDPI, vol. 16(14), pages 1-19, July.
    2. Oke Gerke & Antonia Zapf, 2022. "Convergence Behavior of Optimal Cut-Off Points Derived from Receiver Operating Characteristics Curve Analysis: A Simulation Study," Mathematics, MDPI, vol. 10(22), pages 1-14, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.

      Corrections

      All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:8:y:2020:i:10:p:1741-:d:425988. See general information about how to correct material in RePEc.

      If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

      If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

      If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

      For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

      Please note that corrections may take a couple of weeks to filter through the various RePEc services.

      IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.