IDEAS home Printed from https://ideas.repec.org/a/sae/medema/v43y2023i5p564-575.html
   My bibliography  Save this article

Value-of-Information Analysis for External Validation of Risk Prediction Models

Author

Listed:
  • Mohsen Sadatsafavi

    (Respiratory Evaluation Sciences Program, Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, Canada)

  • Tae Yoon Lee

    (Respiratory Evaluation Sciences Program, Faculty of Pharmaceutical Sciences, University of British Columbia, Vancouver, British Columbia, Canada)

  • Laure Wynants

    (Department of Epidemiology, CAPHRI Care and Public Health Research Institute, Maastricht University, Maastricht, The Netherlands
    Department of Development and Regeneration, KU Leuven, Leuven, Belgium)

  • Andrew J Vickers

    (Department of Epidemiology and Biostatistics, Memorial Sloan Kettering Cancer Center, New York, New York, USA)

  • Paul Gustafson

    (Department of Statistics, University of British Columbia, Vancouver, British Columbia, Canada)

Abstract

Background A previously developed risk prediction model needs to be validated before being used in a new population. The finite size of the validation sample entails that there is uncertainty around model performance. We apply value-of-information (VoI) methodology to quantify the consequence of uncertainty in terms of net benefit (NB). Methods We define the expected value of perfect information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB. We propose bootstrap-based and asymptotic methods for EVPI computations and conduct simulation studies to compare their performance. In a case study, we use the non-US subsets of a clinical trial as the development sample for predicting mortality after myocardial infarction and calculate the validation EVPI for the US subsample. Results The computation methods generated similar EVPI values in simulation studies. EVPI generally declined with larger samples. In the case study, at the prespecified threshold of 0.02, the best decision with current information would be to use the model, with an incremental NB of 0.0020 over treating all. At this threshold, the EVPI was 0.0005 (relative EVPI = 25%). When scaled to the annual number of heart attacks in the US, the expected NB loss due to uncertainty was equal to 400 true positives or 19,600 false positives, indicating the value of further model validation. Conclusion VoI methods can be applied to the NB calculated during external validation of clinical prediction models. While uncertainty does not directly affect the clinical implications of NB findings, validation EVPI provides an objective perspective to the need for further validation and can be reported alongside NB in external validation studies. Highlights External validation is a critical step when transporting a risk prediction model to a new setting, but the finite size of the validation sample creates uncertainty about the performance of the model. In decision theory, such uncertainty is associated with loss of net benefit because it can prevent one from identifying whether the use of the model is beneficial over alternative strategies. We define the expected value of perfect information for external validation as the expected loss in net benefit by not confidently knowing if the use of the model is net beneficial. The adoption of a model for a new population should be based on its expected net benefit; independently, value-of-information methods can be used to decide whether further validation studies are warranted.

Suggested Citation

  • Mohsen Sadatsafavi & Tae Yoon Lee & Laure Wynants & Andrew J Vickers & Paul Gustafson, 2023. "Value-of-Information Analysis for External Validation of Risk Prediction Models," Medical Decision Making, , vol. 43(5), pages 564-575, July.
  • Handle: RePEc:sae:medema:v:43:y:2023:i:5:p:564-575
    DOI: 10.1177/0272989X231178317
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0272989X231178317
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0272989X231178317?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Arellano-Valle, Reinaldo B. & Genton, Marc G., 2008. "On the exact distribution of the maximum of absolutely continuous dependent random variables," Statistics & Probability Letters, Elsevier, vol. 78(1), pages 27-35, January.
    2. Karl Claxton, 1999. "Bayesian approaches to the value of information: implications for the regulation of new pharmaceuticals," Health Economics, John Wiley & Sons, Ltd., vol. 8(3), pages 269-274, May.
    3. James C. Felli & Gordon B. Hazen, 1998. "Sensitivity Analysis and the Expected Value of Perfect Information," Medical Decision Making, , vol. 18(1), pages 95-109, January.
    4. Mohsen Sadatsafavi & Tae Yoon Lee & Paul Gustafson, 2022. "Uncertainty and the Value of Information in Risk Prediction Modeling," Medical Decision Making, , vol. 42(5), pages 661-671, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tae Yoon Lee & Paul Gustafson & Mohsen Sadatsafavi, 2023. "Closed-Form Solution of the Unit Normal Loss Integral in 2 Dimensions, with Application in Value-of-Information Analysis," Medical Decision Making, , vol. 43(5), pages 621-626, July.
    2. Anna Heath & Petros Pechlivanoglou, 2022. "Prioritizing Research in an Era of Personalized Medicine: The Potential Value of Unexplained Heterogeneity," Medical Decision Making, , vol. 42(5), pages 649-660, July.
    3. Oakley, Jeremy E. & Brennan, Alan & Tappenden, Paul & Chilcott, Jim, 2010. "Simulation sample sizes for Monte Carlo partial EVPI calculations," Journal of Health Economics, Elsevier, vol. 29(3), pages 468-477, May.
    4. K. Claxton & P. J. Neumannn & S. S. Araki & M. C. Weinstein, "undated". "Bayesian Value-of-Information Analysis: An Application to a Policy Model of Alzheimer's Disease," Discussion Papers 00/39, Department of Economics, University of York.
    5. Alan Brennan & Samer A. Kharroubi, 2007. "Expected value of sample information for Weibull survival data," Health Economics, John Wiley & Sons, Ltd., vol. 16(11), pages 1205-1225, November.
    6. Alan Brennan & Samer A. Kharroubi, 2007. "Expected value of sample information for Weibull survival data," Health Economics, John Wiley & Sons, Ltd., vol. 16(11), pages 1205-1225.
    7. Claxton, K. & Thompson, K. M., 2001. "A dynamic programming approach to the efficient design of clinical trials," Journal of Health Economics, Elsevier, vol. 20(5), pages 797-822, September.
    8. S. Cucurachi & E. Borgonovo & R. Heijungs, 2016. "A Protocol for the Global Sensitivity Analysis of Impact Assessment Models in Life Cycle Assessment," Risk Analysis, John Wiley & Sons, vol. 36(2), pages 357-377, February.
    9. Nicky Welton & A. E. Ades, 2012. "Research Decisions In The Face Of Heterogeneity: What Can A New Study Tell Us?," Health Economics, John Wiley & Sons, Ltd., vol. 21(10), pages 1196-1200, October.
    10. Edward C. F. Wilson & Miranda Mugford & Garry Barton & Lee Shepstone, 2016. "Efficient Research Design," Medical Decision Making, , vol. 36(3), pages 335-348, April.
    11. Tianyang Wang & James S. Dyer & Warren J. Hahn, 2017. "Sensitivity analysis of decision making under dependent uncertainties using copulas," EURO Journal on Decision Processes, Springer;EURO - The Association of European Operational Research Societies, vol. 5(1), pages 117-139, November.
    12. Jamalizadeh, A. & Balakrishnan, N., 2010. "Distributions of order statistics and linear combinations of order statistics from an elliptical distribution as mixtures of unified skew-elliptical distributions," Journal of Multivariate Analysis, Elsevier, vol. 101(6), pages 1412-1427, July.
    13. Loperfido, Nicola, 2008. "A note on skew-elliptical distributions and linear functions of order statistics," Statistics & Probability Letters, Elsevier, vol. 78(18), pages 3184-3186, December.
    14. Niklas Zethraeus & Magnus Johannesson & Bengt Jönsson & Mickael Löthgren & Magnus Tambour, 2003. "Advantages of Using the Net-Benefit Approach for Analysing Uncertainty in Economic Evaluation Studies," PharmacoEconomics, Springer, vol. 21(1), pages 39-48, January.
    15. Martin Henriksson & Fredrik Lundgren & Per Carlsson, 2006. "Informing the efficient use of health care and health care research resources ‐ the case of screening for abdominal aortic aneurysm in Sweden," Health Economics, John Wiley & Sons, Ltd., vol. 15(12), pages 1311-1322, December.
    16. Rachael L. Fleurence, 2007. "Setting priorities for research: a practical application of 'payback' and expected value of information," Health Economics, John Wiley & Sons, Ltd., vol. 16(12), pages 1345-1357.
    17. Claire McKenna & Karl Claxton, 2011. "Addressing Adoption and Research Design Decisions Simultaneously," Medical Decision Making, , vol. 31(6), pages 853-865, November.
    18. Heikkinen, T. & Pietola, K., 2009. "Investment and the dynamic cost of income uncertainty: The case of diminishing expectations in agriculture," European Journal of Operational Research, Elsevier, vol. 192(2), pages 634-646, January.
    19. Stefano Conti & Karl Claxton, 2008. "Dimensions of design space: a decision-theoretic approach to optimal research design," Working Papers 038cherp, Centre for Health Economics, University of York.
    20. Doug Coyle & Jeremy Oakley, 2008. "Estimating the expected value of partial perfect information: a review of methods," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 9(3), pages 251-259, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:medema:v:43:y:2023:i:5:p:564-575. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.