IDEAS home Printed from https://ideas.repec.org/a/sae/medema/v44y2024i1p53-63.html
   My bibliography  Save this article

Evaluating Risk Prediction with Data Collection Costs: Novel Estimation of Test Tradeoff Curves

Author

Listed:
  • Stuart G. Baker

Abstract

Background The test tradeoff curve helps investigators decide if collecting data for risk prediction is worthwhile when risk prediction is used for treatment decisions. At a given benefit-cost ratio (the number of false-positive predictions one would trade for a true positive prediction) or risk threshold (the probability of developing disease at indifference between treatment and no treatment), the test tradeoff is the minimum number of data collections per true positive to yield a positive maximum expected utility of risk prediction. For example, a test tradeoff of 3,000 invasive tests per true-positive prediction of cancer may suggest that risk prediction is not worthwhile. A test tradeoff curve plots test tradeoff versus benefit-cost ratio or risk threshold. The test tradeoff curve evaluates risk prediction at the optimal risk score cutpoint for treatment, which is the cutpoint of the risk score (the estimated risk of developing disease) that maximizes the expected utility of risk prediction when the receiver-operating characteristic (ROC) curve is concave. Methods Previous methods for estimating the test tradeoff required grouping risk scores. Using individual risk scores, the new method estimates a concave ROC curve by constructing a concave envelope of ROC points, taking a slope-based moving average, minimizing a sum of squared errors, and connecting successive ROC points with line segments. Results The estimated concave ROC curve yields an estimated test tradeoff curve. Analyses of 2 synthetic data sets illustrate the method. Conclusion Estimating the test tradeoff curve based on individual risk scores is straightforward to implement and more appealing than previous estimation methods that required grouping risk scores. Highlights The test tradeoff curve helps investigators decide if collecting data for risk prediction is worthwhile when risk prediction is used for treatment decisions. At a given benefit-cost ratio or risk threshold, the test tradeoff is the minimum number of data collections per true positive to yield a positive maximum expected utility of risk prediction. Unlike previous estimation methods that grouped risk scores, the method uses individual risk scores to estimate a concave ROC curve, which yields an estimated test tradeoff curve.

Suggested Citation

  • Stuart G. Baker, 2024. "Evaluating Risk Prediction with Data Collection Costs: Novel Estimation of Test Tradeoff Curves," Medical Decision Making, , vol. 44(1), pages 53-63, January.
  • Handle: RePEc:sae:medema:v:44:y:2024:i:1:p:53-63
    DOI: 10.1177/0272989X231208673
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0272989X231208673
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0272989X231208673?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Stuart G. Baker, 2018. "Simple Decision-Analytic Functions of the AUC for Ruling Out a Risk Prediction Model and an Added Predictor," Medical Decision Making, , vol. 38(2), pages 225-234, February.
    2. Stuart G. Baker & Nancy R. Cook & Andrew Vickers & Barnett S. Kramer, 2009. "Using relative utility curves to evaluate risk prediction," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(4), pages 729-748, October.
    3. Andrew J. Vickers & Elena B. Elkin, 2006. "Decision Curve Analysis: A Novel Method for Evaluating Prediction Models," Medical Decision Making, , vol. 26(6), pages 565-574, November.
    4. Kathleen F. Kerr & Marshall D. Brown & Tracey L. Marsh & Holly Janes, 2019. "Assessing the Clinical Impact of Risk Models for Opting Out of Treatment," Medical Decision Making, , vol. 39(2), pages 86-90, February.
    5. Natalia Olchanski & Joshua T. Cohen & Peter J. Neumann & John B. Wong & David M. Kent, 2017. "Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models," Medical Decision Making, , vol. 37(7), pages 790-801, October.
    6. Stuart G. Baker & Kurt Heidenberger, 1989. "Choosing Sample Sizes to Maximize Expected Health Benefits Subject to a Constraint on Total Trial Costs," Medical Decision Making, , vol. 9(1), pages 14-25, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tracey L. Marsh & Holly Janes & Margaret S. Pepe, 2020. "Statistical inference for net benefit measures in biomarker validation studies," Biometrics, The International Biometric Society, vol. 76(3), pages 843-852, September.
    2. Baker Stuart G. & Van Calster Ben & Steyerberg Ewout W., 2012. "Evaluating a New Marker for Risk Prediction Using the Test Tradeoff: An Update," The International Journal of Biostatistics, De Gruyter, vol. 8(1), pages 1-37, March.
    3. Ying Huang & Eric Laber, 2016. "Personalized Evaluation of Biomarker Value: A Cost-Benefit Perspective," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 8(1), pages 43-65, June.
    4. Hormuzd A. Katki & Ionut Bebu, 2021. "A simple framework to identify optimal cost‐effective risk thresholds for a single screen: Comparison to Decision Curve Analysis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(3), pages 887-903, July.
    5. Ja Hyeon Ku & Myong Kim & Seok-Soo Byun & Hyeon Jeong & Cheol Kwak & Hyeon Hoe Kim & Sang Eun Lee, 2015. "External Validation of Models for Prediction of Lymph Node Metastasis in Urothelial Carcinoma of the Bladder," PLOS ONE, Public Library of Science, vol. 10(10), pages 1-10, October.
    6. Lin Lu & Laurent Dercle & Binsheng Zhao & Lawrence H. Schwartz, 2021. "Deep learning for the prediction of early on-treatment response in metastatic colorectal cancer from serial medical imaging," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    7. Yiwang Zhou & Peter X.K. Song & Haoda Fu, 2021. "Net benefit index: Assessing the influence of a biomarker for individualized treatment rules," Biometrics, The International Biometric Society, vol. 77(4), pages 1254-1264, December.
    8. Jing Sun & Yue Liu & Jianhui Zhao & Bin Lu & Siyun Zhou & Wei Lu & Jingsun Wei & Yeting Hu & Xiangxing Kong & Junshun Gao & Hong Guan & Junli Gao & Qian Xiao & Xue Li, 2024. "Plasma proteomic and polygenic profiling improve risk stratification and personalized screening for colorectal cancer," Nature Communications, Nature, vol. 15(1), pages 1-10, December.
    9. Ben Van Calster & Ewout W. Steyerberg & Ralph B. D’Agostino Sr & Michael J. Pencina, 2014. "Sensitivity and Specificity Can Change in Opposite Directions When New Predictive Markers Are Added to Risk Models," Medical Decision Making, , vol. 34(4), pages 513-522, May.
    10. Shamil D. Cooray & Lihini A. Wijeyaratne & Georgia Soldatos & John Allotey & Jacqueline A. Boyle & Helena J. Teede, 2020. "The Unrealised Potential for Predicting Pregnancy Complications in Women with Gestational Diabetes: A Systematic Review and Critical Appraisal," IJERPH, MDPI, vol. 17(9), pages 1-20, April.
    11. Khushal Arjan & Lui G Forni & Richard M Venn & David Hunt & Luke Eliot Hodgson, 2021. "Clinical decision-making in older adults following emergency admission to hospital. Derivation and validation of a risk stratification score: OPERA," PLOS ONE, Public Library of Science, vol. 16(3), pages 1-12, March.
    12. Christian Bock & Joan Elias Walter & Bastian Rieck & Ivo Strebel & Klara Rumora & Ibrahim Schaefer & Michael J. Zellweger & Karsten Borgwardt & Christian Müller, 2024. "Enhancing the diagnosis of functionally relevant coronary artery disease with machine learning," Nature Communications, Nature, vol. 15(1), pages 1-16, December.
    13. Chang Wook Jeong & Sangchul Lee & Jin-Woo Jung & Byung Ki Lee & Seong Jin Jeong & Sung Kyu Hong & Seok-Soo Byun & Sang Eun Lee, 2014. "Mobile Application-Based Seoul National University Prostate Cancer Risk Calculator: Development, Validation, and Comparative Analysis with Two Western Risk Calculators in Korean Men," PLOS ONE, Public Library of Science, vol. 9(4), pages 1-7, April.
    14. Shi, Chengchun & Lu, Wenbin & Song, Rui, 2019. "A sparse random projection-based test for overall qualitative treatment effects," LSE Research Online Documents on Economics 102107, London School of Economics and Political Science, LSE Library.
    15. Yanqing Wang & Yingqi Zhao & Yingye Zheng, 2022. "Targeted Search for Individualized Clinical Decision Rules to Optimize Clinical Outcomes," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 14(3), pages 564-581, December.
    16. Tae Yoon Lee & Paul Gustafson & Mohsen Sadatsafavi, 2023. "Closed-Form Solution of the Unit Normal Loss Integral in 2 Dimensions, with Application in Value-of-Information Analysis," Medical Decision Making, , vol. 43(5), pages 621-626, July.
    17. Mei-Cheng Wang & Shanshan Li, 2012. "Bivariate Marker Measurements and ROC Analysis," Biometrics, The International Biometric Society, vol. 68(4), pages 1207-1218, December.
    18. Holly Janes & Margaret S. Pepe & Ying Huang, 2014. "A Framework for Evaluating Markers Used to Select Patient Treatment," Medical Decision Making, , vol. 34(2), pages 159-167, February.
    19. Kevin Sandeman & Juho T Eineluoto & Joona Pohjonen & Andrew Erickson & Tuomas P Kilpeläinen & Petrus Järvinen & Henrikki Santti & Anssi Petas & Mika Matikainen & Suvi Marjasuo & Anu Kenttämies & Tuoma, 2020. "Prostate MRI added to CAPRA, MSKCC and Partin cancer nomograms significantly enhances the prediction of adverse findings and biochemical recurrence after radical prostatectomy," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-14, July.
    20. Danny J N Wong & Steve Harris & Arun Sahni & James R Bedford & Laura Cortes & Richard Shawyer & Andrew M Wilson & Helen A Lindsay & Doug Campbell & Scott Popham & Lisa M Barneto & Paul S Myles & SNAP-, 2020. "Developing and validating subjective and objective risk-assessment measures for predicting mortality after major surgery: An international prospective cohort study," PLOS Medicine, Public Library of Science, vol. 17(10), pages 1-22, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:medema:v:44:y:2024:i:1:p:53-63. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.