IDEAS home Printed from https://ideas.repec.org/a/sae/medema/v37y2017i7p790-801.html
   My bibliography  Save this article

Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models

Author

Listed:
  • Natalia Olchanski
  • Joshua T. Cohen
  • Peter J. Neumann
  • John B. Wong
  • David M. Kent

Abstract

Background. Risk prediction models allow for the incorporation of individualized risk and clinical effectiveness information to identify patients for whom therapy is most appropriate and cost-effective. This approach has the potential to identify inefficient (or harmful) care in subgroups at different risks, even when the overall results appear favorable. Here, we explore the value of personalized risk information and the factors that influence it. Methods. Using an expected value of individualized care (EVIC) framework, which monetizes the value of customizing care, we developed a general approach to calculate individualized incremental cost effectiveness ratios (ICERs) as a function of individual outcome risk. For a case study (tPA v. streptokinase to treat possible myocardial infarction), we used a simulation to explore how an EVIC is influenced by population outcome prevalence, model discrimination (c-statistic) and calibration, and willingness-to-pay (WTP) thresholds. Results. In our simulations, for well-calibrated models, which do not over- or underestimate predicted v. observed event risk, the EVIC ranged from $0 to $700 per person, with better discrimination (higher c-statistic values) yielding progressively higher EVIC values. For miscalibrated models, the EVIC ranged from −$600 to $600 in different simulated scenarios. The EVIC values decreased as discrimination improved from a c-statistic of 0.5 to 0.6, before becoming positive as the c-statistic reached values of ~0.8. Conclusions. Individualizing treatment decisions using risk may produce substantial value but also has the potential for net harm. Good model calibration ensures a non-negative EVIC. Improvements in discrimination generally increase the EVIC; however, when models are miscalibrated, greater discriminating power can paradoxically reduce the EVIC under some circumstances.

Suggested Citation

  • Natalia Olchanski & Joshua T. Cohen & Peter J. Neumann & John B. Wong & David M. Kent, 2017. "Understanding the Value of Individualized Information: The Impact of Poor Calibration or Discrimination in Outcome Prediction Models," Medical Decision Making, , vol. 37(7), pages 790-801, October.
  • Handle: RePEc:sae:medema:v:37:y:2017:i:7:p:790-801
    DOI: 10.1177/0272989X17704855
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0272989X17704855
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0272989X17704855?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Meltzer, David, 2001. "Addressing uncertainty in medical cost-effectiveness analysis: Implications of expected utility maximization for methods to perform sensitivity analysis and the use of cost-effectiveness analysis to s," Journal of Health Economics, Elsevier, vol. 20(1), pages 109-129, January.
    2. Claxton, Karl, 1999. "The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies," Journal of Health Economics, Elsevier, vol. 18(3), pages 341-364, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stuart G. Baker, 2024. "Evaluating Risk Prediction with Data Collection Costs: Novel Estimation of Test Tradeoff Curves," Medical Decision Making, , vol. 44(1), pages 53-63, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Charles F. Manski, 2022. "Patient‐centered appraisal of race‐free clinical risk assessment," Health Economics, John Wiley & Sons, Ltd., vol. 31(10), pages 2109-2114, October.
    2. Rachael L. Fleurence, 2007. "Setting priorities for research: a practical application of 'payback' and expected value of information," Health Economics, John Wiley & Sons, Ltd., vol. 16(12), pages 1345-1357.
    3. Samer A. Kharroubi & Alan Brennan & Mark Strong, 2011. "Estimating Expected Value of Sample Information for Incomplete Data Models Using Bayesian Approximation," Medical Decision Making, , vol. 31(6), pages 839-852, November.
    4. Anirban Basu & David Meltzer, 2012. "Private Manufacturers’ Thresholds to Invest in Comparative Effectiveness Trials," PharmacoEconomics, Springer, vol. 30(10), pages 859-868, October.
    5. Fleurence, Rachael L. & Torgerson, David J., 2004. "Setting priorities for research," Health Policy, Elsevier, vol. 69(1), pages 1-10, July.
    6. Brennan, Alan & Kharroubi, Samer A., 2007. "Efficient computation of partial expected value of sample information using Bayesian approximation," Journal of Health Economics, Elsevier, vol. 26(1), pages 122-148, January.
    7. Alan Brennan & Samer A. Kharroubi, 2007. "Expected value of sample information for Weibull survival data," Health Economics, John Wiley & Sons, Ltd., vol. 16(11), pages 1205-1225, November.
    8. Alan Brennan & Samer Kharroubi & Anthony O'Hagan & Jim Chilcott, 2007. "Calculating Partial Expected Value of Perfect Information via Monte Carlo Sampling Algorithms," Medical Decision Making, , vol. 27(4), pages 448-470, July.
    9. Eldon Spackman & Stewart Richmond & Mark Sculpher & Martin Bland & Stephen Brealey & Rhian Gabe & Ann Hopton & Ada Keding & Harriet Lansdown & Sara Perren & David Torgerson & Ian Watt & Hugh MacPherso, 2014. "Cost-Effectiveness Analysis of Acupuncture, Counselling and Usual Care in Treating Patients with Depression: The Results of the ACUDep Trial," PLOS ONE, Public Library of Science, vol. 9(11), pages 1-12, November.
    10. Qi Cao & Erik Buskens & Hans L. Hillege & Tiny Jaarsma & Maarten Postma & Douwe Postmus, 2019. "Stratified treatment recommendation or one-size-fits-all? A health economic insight based on graphical exploration," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(3), pages 475-482, April.
    11. Thomas Reinhold & Claudia Witt & Susanne Jena & Benno Brinkhaus & Stefan Willich, 2008. "Quality of life and cost-effectiveness of acupuncture treatment in patients with osteoarthritis pain," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 9(3), pages 209-219, August.
    12. Isaac Corro Ramos & Maureen P. M. H. Rutten-van Mölken & Maiwenn J. Al, 2013. "The Role of Value-of-Information Analysis in a Health Care Research Priority Setting," Medical Decision Making, , vol. 33(4), pages 472-489, May.
    13. Maiwenn Al, 2013. "Cost-Effectiveness Acceptability Curves Revisited," PharmacoEconomics, Springer, vol. 31(2), pages 93-100, February.
    14. Katharina Fischer & Reiner Leidl, 2014. "Analysing coverage decision-making: opening Pandora’s box?," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 15(9), pages 899-906, December.
    15. Mark Sculpher & Amiram Gafni, 2001. "Recognizing diversity in public preferences: The use of preference sub‐groups in cost‐effectiveness analysis," Health Economics, John Wiley & Sons, Ltd., vol. 10(4), pages 317-324, June.
    16. McKenna, Claire & Chalabi, Zaid & Epstein, David & Claxton, Karl, 2010. "Budgetary policies and available actions: A generalisation of decision rules for allocation and research decisions," Journal of Health Economics, Elsevier, vol. 29(1), pages 170-181, January.
    17. Niklas Zethraeus & Magnus Johannesson & Bengt Jönsson & Mickael Löthgren & Magnus Tambour, 2003. "Advantages of Using the Net-Benefit Approach for Analysing Uncertainty in Economic Evaluation Studies," PharmacoEconomics, Springer, vol. 21(1), pages 39-48, January.
    18. Wilding, Sarah & Tsipa, Anastasia & Branley-Bell, Dawn & Greenwood, Darren C. & Vargas-Palacios, Armando & Yaziji, Nahel & Addison, Caroline & Kelly, Phil & Day, Fiona & Horsfall, Kate & Conner, Mark , 2020. "Cluster randomized controlled trial of volitional and motivational interventions to improve bowel cancer screening uptake: A population-level study," Social Science & Medicine, Elsevier, vol. 265(C).
    19. Claire McKenna & Karl Claxton, 2011. "Addressing Adoption and Research Design Decisions Simultaneously," Medical Decision Making, , vol. 31(6), pages 853-865, November.
    20. Nadia Demarteau & Thomas Breuer & Baudouin Standaert, 2012. "Selecting a Mix of Prevention Strategies against Cervical Cancer for Maximum Efficiency with an Optimization Program," PharmacoEconomics, Springer, vol. 30(4), pages 337-353, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:medema:v:37:y:2017:i:7:p:790-801. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.