Is Separately Modeling Subpopulations Beneficial for Sequential Decision-Making?

In recent applications of Markov decision processes (MDPs), it is common to estimate transition probabilities and rewards from transition data. In healthcare and some other applications, transition data are collected from a population of different entities, such as patients. Thus, one faces a modeling question of whether to estimate different models for subpopulations (e.g., divided by smoking status). For instance, there may be a subpopulation whose disease status progresses faster than others, and for such a group, estimating a separate model and applying the corresponding optimal treatment plan can improve their outcomes. This work provides theoretical results and empirical methods for making the decision of whether to model subpopulations separately (called stratifying) or not. We also present how to use our results to select the best stratification among many. We illustrate our results and methods numerically using random instances and a medical decision-making problem from the literature. Because improving medical decisions by tailoring to each subpopulation is a building block of precision medicine, this work advances the use of MDPs in medical decision-making toward the precision medicine paradigm.