A robust threshold t linear mixed model for subgroup identification using multivariate T distributions

Subgroup identification has emerged as a popular statistical tool to access the heterogeneity in treatment effects based on specific characteristics of patients. Recently, a threshold linear mixed-effects model was proposed to identify a subgroup of patients who may benefit from treatment concerning longitudinal outcomes based on whether a continuous biomarker exceeds an unknown cut-point. This model assumes, however, normal distributions to both random effects and error terms and, therefore, may be sensitive to outliers in the longitudinal outcomes. In this paper, we propose a robust subgroup identification method for longitudinal data by developing a robust threshold t linear mixed-effects model, where random effects and within-subject errors follow a multivariate t distribution, with unknown degree-of-freedoms. The likelihood function is, however, difficult to directly maximize because the density function of a non-central t distribution is too complicated to compute and an indicator function is included in the definition of the mode. We, therefore, propose a smoothed expectation conditional maximization algorithm based on a gamma-normal hierarchical structure and the smooth approximation of an indicator function to make inferences on the parameters in the model. Simulation studies are conducted to investigate the performance and robustness of the proposed method. As an application, the proposed method is used to identify a subgroup of patients with advanced colorectal cancer who may have a better quality of life when treated by cetuximab.