IDEAS home Printed from https://ideas.repec.org/a/sae/medema/v33y2013i4p490-501.html
   My bibliography  Save this article

Evaluation of Markers and Risk Prediction Models

Author

Listed:
  • Ben Van Calster
  • Andrew J. Vickers
  • Michael J. Pencina
  • Stuart G. Baker
  • Dirk Timmerman
  • Ewout W. Steyerberg

Abstract

Background . For the evaluation and comparison of markers and risk prediction models, various novel measures have recently been introduced as alternatives to the commonly used difference in the area under the receiver operating characteristic (ROC) curve (ΔAUC). The net reclassification improvement (NRI) is increasingly popular to compare predictions with 1 or more risk thresholds, but decision-analytic approaches have also been proposed. Objective . We aimed to identify the mathematical relationships between novel performance measures for the situation that a single risk threshold T is used to classify patients as having the outcome or not. Methods . We considered the NRI and 3 utility-based measures that take misclassification costs into account: difference in net benefit (ΔNB), difference in relative utility (ΔRU), and weighted NRI (wNRI). We illustrate the behavior of these measures in 1938 women suspect of having ovarian cancer (prevalence 28%). Results . The 3 utility-based measures appear to be transformations of each other and hence always lead to consistent conclusions. On the other hand, conclusions may differ when using the standard NRI, depending on the adopted risk threshold T , prevalence P , and the obtained differences in sensitivity and specificity of the 2 models that are compared. In the case study, adding the CA-125 tumor marker to a baseline set of covariates yielded a negative NRI yet a positive value for the utility-based measures. Conclusions . The decision-analytic measures are each appropriate to indicate the clinical usefulness of an added marker or compare prediction models since these measures each reflect misclassification costs. This is of practical importance as these measures may thus adjust conclusions based on purely statistical measures. A range of risk thresholds should be considered in applying these measures.

Suggested Citation

  • Ben Van Calster & Andrew J. Vickers & Michael J. Pencina & Stuart G. Baker & Dirk Timmerman & Ewout W. Steyerberg, 2013. "Evaluation of Markers and Risk Prediction Models," Medical Decision Making, , vol. 33(4), pages 490-501, May.
    • Handle: RePEc:sae:medema:v:33:y:2013:i:4:p:490-501
    DOI: 10.1177/0272989X12470757
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0272989X12470757
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0272989X12470757?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Baker, Stuart G. & Kramer, Barnett S., 2007. "Peirce, Youden, and Receiver Operating Characteristic Curves," The American Statistician, American Statistical Association, vol. 61, pages 343-346, November.
    2. Stuart G. Baker & Nancy R. Cook & Andrew Vickers & Barnett S. Kramer, 2009. "Using relative utility curves to evaluate risk prediction," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(4), pages 729-748, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ben Van Calster & Andrew J. Vickers, 2015. "Calibration of Risk Prediction Models," Medical Decision Making, , vol. 35(2), pages 162-169, February.
    2. Bonato, Matteo & Demirer, Riza & Gupta, Rangan & Pierdzioch, Christian, 2018. "Gold futures returns and realized moments: A forecasting experiment using a quantile-boosting approach," Resources Policy, Elsevier, vol. 57(C), pages 196-212.
    3. Döpke, Jörg & Fritsche, Ulrich & Pierdzioch, Christian, 2017. "Predicting recessions with boosted regression trees," International Journal of Forecasting, Elsevier, vol. 33(4), pages 745-759.
    4. Travis J. Berge & Shu-Chun Chen & Hsieh Fushing & Òscar Jordà, 2010. "A chronology of international business cycles through non-parametric decoding," Research Working Paper RWP 11-13, Federal Reserve Bank of Kansas City.
    5. Ben Van Calster & Ewout W. Steyerberg & Ralph B. D’Agostino Sr & Michael J. Pencina, 2014. "Sensitivity and Specificity Can Change in Opposite Directions When New Predictive Markers Are Added to Risk Models," Medical Decision Making, , vol. 34(4), pages 513-522, May.
    6. Stuart G. Baker, 2019. "Cancer Screening Markers: A Simple Strategy to Substantially Reduce the Sample Size for Validation," Medical Decision Making, , vol. 39(2), pages 130-136, February.
    7. Travis Berge & Òscar Jordà, 2013. "A chronology of turning points in economic activity: Spain, 1850–2011," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 4(1), pages 1-34, March.
    8. Pierdzioch, Christian & Rülke, Jan-Christoph, 2015. "On the directional accuracy of forecasts of emerging market exchange rates," International Review of Economics & Finance, Elsevier, vol. 38(C), pages 369-376.
    9. Drehmann, Mathias & Juselius, Mikael, 2014. "Evaluating early warning indicators of banking crises: Satisfying policy requirements," International Journal of Forecasting, Elsevier, vol. 30(3), pages 759-780.
    10. Yusuf Yıldırım & Anirban Sanyal, 2022. "Evaluating the Effectiveness of Early Warning Indicators: An Application of Receiver Operating Characteristic Curve Approach to Panel Data," Scientific Annals of Economics and Business (continues Analele Stiintifice), Alexandru Ioan Cuza University, Faculty of Economics and Business Administration, vol. 69(4), pages 557-597, December.
    11. Tracey L. Marsh & Holly Janes & Margaret S. Pepe, 2020. "Statistical inference for net benefit measures in biomarker validation studies," Biometrics, The International Biometric Society, vol. 76(3), pages 843-852, September.
    12. Shi, Chengchun & Lu, Wenbin & Song, Rui, 2019. "A sparse random projection-based test for overall qualitative treatment effects," LSE Research Online Documents on Economics 102107, London School of Economics and Political Science, LSE Library.
    13. Hormuzd A. Katki & Ionut Bebu, 2021. "A simple framework to identify optimal cost‐effective risk thresholds for a single screen: Comparison to Decision Curve Analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(3), pages 887-903, July.
    14. Stuart G. Baker, 2024. "Evaluating Risk Prediction with Data Collection Costs: Novel Estimation of Test Tradeoff Curves," Medical Decision Making, , vol. 44(1), pages 53-63, January.
    15. Qing Lu & Nancy Obuchowski & Sungho Won & Xiaofeng Zhu & Robert C. Elston, 2010. "Using the Optimal Robust Receiver Operating Characteristic (ROC) Curve for Predictive Genetic Tests," Biometrics, The International Biometric Society, vol. 66(2), pages 586-593, June.
    16. Mei-Cheng Wang & Shanshan Li, 2012. "Bivariate Marker Measurements and ROC Analysis," Biometrics, The International Biometric Society, vol. 68(4), pages 1207-1218, December.
    17. Holly Janes & Margaret S. Pepe & Ying Huang, 2014. "A Framework for Evaluating Markers Used to Select Patient Treatment," Medical Decision Making, , vol. 34(2), pages 159-167, February.
    18. Scott Brave & R. Andrew Butters, 2014. "Nowcasting Using the Chicago Fed National Activity Index," Economic Perspectives, Federal Reserve Bank of Chicago, issue Q I, pages 19-37.
    19. Baker Stuart G. & Van Calster Ben & Steyerberg Ewout W., 2012. "Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update," The International Journal of Biostatistics, De Gruyter, vol. 8(1), pages 1-37, March.
    20. Pierdzioch Christian & Gupta Rangan, 2020. "Uncertainty and Forecasts of U.S. Recessions," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 24(4), pages 1-20, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:medema:v:33:y:2013:i:4:p:490-501. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.