IDEAS home Printed from https://ideas.repec.org/a/gam/jstats/v4y2021i4p46-792d643999.html
   My bibliography  Save this article

Survival Augmented Patient Preference Incorporated Reinforcement Learning to Evaluate Tailoring Variables for Personalized Healthcare

Author

Listed:
  • Yingchao Zhong

    (Department of Biostatistics, University of Michigan, Ann Arbor, MI 48109, USA)

  • Chang Wang

    (Department of Biostatistics, University of Michigan, Ann Arbor, MI 48109, USA)

  • Lu Wang

    (Department of Biostatistics, University of Michigan, Ann Arbor, MI 48109, USA)

Abstract

In this paper, we consider personalized treatment decision strategies in the management of chronic diseases, such as chronic kidney disease, which typically consists of sequential and adaptive treatment decision making. We investigate a two-stage treatment setting with a survival outcome that could be right censored. This can be formulated through a dynamic treatment regime (DTR) framework, where the goal is to tailor treatment to each individual based on their own medical history in order to maximize a desirable health outcome. We develop a new method, Survival Augmented Patient Preference incorporated reinforcement Q-Learning (SAPP-Q-Learning) to decide between quality of life and survival restricted at maximal follow-up. Our method incorporates the latent patient preference into a weighted utility function that balances between quality of life and survival time, in a Q-learning model framework. We further propose a corresponding m-out-of-n Bootstrap procedure to accurately make statistical inferences and construct confidence intervals on the effects of tailoring variables, whose values can guide personalized treatment strategies.

Suggested Citation

  • Yingchao Zhong & Chang Wang & Lu Wang, 2021. "Survival Augmented Patient Preference Incorporated Reinforcement Learning to Evaluate Tailoring Variables for Personalized Healthcare," Stats, MDPI, vol. 4(4), pages 1-17, September.
  • Handle: RePEc:gam:jstats:v:4:y:2021:i:4:p:46-792:d:643999
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2571-905X/4/4/46/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2571-905X/4/4/46/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. E. B. Laber & Y. Q. Zhao, 2015. "Tree-based methods for individualized treatment regimes," Biometrika, Biometrika Trust, vol. 102(3), pages 501-514.
    2. Yichi Zhang & Eric B. Laber & Marie Davidian & Anastasios A. Tsiatis, 2018. "Interpretable Dynamic Treatment Regimes," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1541-1549, October.
    3. Yebin Tao & Lu Wang, 2017. "Adaptive contrast weighted learning for multi-stage multi-treatment decision-making," Biometrics, The International Biometric Society, vol. 73(1), pages 145-155, March.
    4. Bibhas Chakraborty & Eric B. Laber & Yingqi Zhao, 2013. "Inference for Optimal Dynamic Treatment Regimes Using an Adaptive m-Out-of-n Bootstrap Scheme," Biometrics, The International Biometric Society, vol. 69(3), pages 714-723, September.
    5. S. A. Murphy, 2003. "Optimal dynamic treatment regimes," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 65(2), pages 331-355, May.
    6. Rebecca Hager & Anastasios A. Tsiatis & Marie Davidian, 2018. "Optimal two‐stage dynamic treatment regimes from a classification perspective with censored survival data," Biometrics, The International Biometric Society, vol. 74(4), pages 1180-1192, December.
    7. Yilun Sun & Lu Wang, 2021. "Stochastic Tree Search for Estimating Optimal Dynamic Treatment Regimes," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(533), pages 421-432, January.
    8. Emily L. Butler & Eric B. Laber & Sonia M. Davis & Michael R. Kosorok, 2018. "Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules," Biometrics, The International Biometric Society, vol. 74(1), pages 18-26, March.
    9. Yingqi Zhao & Donglin Zeng & A. John Rush & Michael R. Kosorok, 2012. "Estimating Individualized Treatment Rules Using Outcome Weighted Learning," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(499), pages 1106-1118, September.
    10. Y. Q. Zhao & D. Zeng & E. B. Laber & R. Song & M. Yuan & M. R. Kosorok, 2015. "Doubly robust learning for estimating individualized treatment with censored data," Biometrika, Biometrika Trust, vol. 102(1), pages 151-168.
    11. Baqun Zhang & Min Zhang, 2018. "C‐learning: A new classification framework to estimate optimal dynamic treatment regimes," Biometrics, The International Biometric Society, vol. 74(3), pages 891-899, September.
    12. Irini Moustaki & Martin Knott, 2000. "Generalized latent trait models," Psychometrika, Springer;The Psychometric Society, vol. 65(3), pages 391-411, September.
    13. Ruoqing Zhu & Michael R. Kosorok, 2012. "Recursively Imputed Survival Trees," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(497), pages 331-340, March.
    14. Runchao Jiang & Wenbin Lu & Rui Song & Marie Davidian, 2017. "On estimation of optimal treatment regimes for maximizing t-year survival probability," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(4), pages 1165-1185, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tian Zhu & Wei Zhu, 2022. "Quantitative Trading through Random Perturbation Q-Network with Nonlinear Transaction Costs," Stats, MDPI, vol. 5(2), pages 1-15, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dana Johnson & Wenbin Lu & Marie Davidian, 2023. "A general framework for subgroup detection via one‐step value difference estimation," Biometrics, The International Biometric Society, vol. 79(3), pages 2116-2126, September.
    2. Yunan Wu & Lan Wang, 2021. "Resampling‐based confidence intervals for model‐free robust inference on optimal treatment regimes," Biometrics, The International Biometric Society, vol. 77(2), pages 465-476, June.
    3. Shuxiao Chen & Bo Zhang, 2021. "Estimating and Improving Dynamic Treatment Regimes With a Time-Varying Instrumental Variable," Papers 2104.07822, arXiv.org.
    4. Q. Clairon & R. Henderson & N. J. Young & E. D. Wilson & C. J. Taylor, 2021. "Adaptive treatment and robust control," Biometrics, The International Biometric Society, vol. 77(1), pages 223-236, March.
    5. Jin Wang & Donglin Zeng & D. Y. Lin, 2022. "Semiparametric single-index models for optimal treatment regimens with censored outcomes," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 28(4), pages 744-763, October.
    6. Giorgos Bakoyannis, 2023. "Estimating optimal individualized treatment rules with multistate processes," Biometrics, The International Biometric Society, vol. 79(4), pages 2830-2842, December.
    7. Hongming Pu & Bo Zhang, 2021. "Estimating optimal treatment rules with an instrumental variable: A partial identification learning approach," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(2), pages 318-345, April.
    8. Zhang, Haixiang & Huang, Jian & Sun, Liuquan, 2020. "A rank-based approach to estimating monotone individualized two treatment regimes," Computational Statistics & Data Analysis, Elsevier, vol. 151(C).
    9. Zhen Li & Jie Chen & Eric Laber & Fang Liu & Richard Baumgartner, 2023. "Optimal Treatment Regimes: A Review and Empirical Comparison," International Statistical Review, International Statistical Institute, vol. 91(3), pages 427-463, December.
    10. Lingyun Lyu & Yu Cheng & Abdus S. Wahed, 2023. "Imputation‐based Q‐learning for optimizing dynamic treatment regimes with right‐censored survival outcome," Biometrics, The International Biometric Society, vol. 79(4), pages 3676-3689, December.
    11. Shosei Sakaguchi, 2024. "Robust Learning for Optimal Dynamic Treatment Regimes with Observational Data," Papers 2404.00221, arXiv.org.
    12. Kara E. Rudolph & Iván Díaz, 2022. "When the ends do not justify the means: Learning who is predicted to have harmful indirect effects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(S2), pages 573-589, December.
    13. Yaoyao Xu & Menggang Yu & Ying‐Qi Zhao & Quefeng Li & Sijian Wang & Jun Shao, 2015. "Regularized outcome weighted subgroup identification for differential treatment effects," Biometrics, The International Biometric Society, vol. 71(3), pages 645-653, September.
    14. Xin Chen & Rui Song & Jiajia Zhang & Swann Arp Adams & Liuquan Sun & Wenbin Lu, 2022. "On estimating optimal regime for treatment initiation time based on restricted mean residual lifetime," Biometrics, The International Biometric Society, vol. 78(4), pages 1377-1389, December.
    15. Yanqing Wang & Yingqi Zhao & Yingye Zheng, 2022. "Targeted Search for Individualized Clinical Decision Rules to Optimize Clinical Outcomes," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 14(3), pages 564-581, December.
    16. Nina Zhou & Lu Wang & Daniel Almirall, 2023. "Estimating tree‐based dynamic treatment regimes using observational data with restricted treatment sequences," Biometrics, The International Biometric Society, vol. 79(3), pages 2260-2271, September.
    17. Michael P. Wallace & Erica E. M. Moodie, 2015. "Doubly‐robust dynamic treatment regimen estimation via weighted least squares," Biometrics, The International Biometric Society, vol. 71(3), pages 636-644, September.
    18. Runchao Jiang & Wenbin Lu & Rui Song & Marie Davidian, 2017. "On estimation of optimal treatment regimes for maximizing t-year survival probability," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 79(4), pages 1165-1185, September.
    19. Emily L. Butler & Eric B. Laber & Sonia M. Davis & Michael R. Kosorok, 2018. "Incorporating Patient Preferences into Estimation of Optimal Individualized Treatment Rules," Biometrics, The International Biometric Society, vol. 74(1), pages 18-26, March.
    20. Hyung G. Park & Danni Wu & Eva Petkova & Thaddeus Tarpey & R. Todd Ogden, 2023. "Bayesian Index Models for Heterogeneous Treatment Effects on a Binary Outcome," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 15(2), pages 397-418, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jstats:v:4:y:2021:i:4:p:46-792:d:643999. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.