Bayesian criterion based model assessment for categorical data

We propose a general Bayesian criterion for model assessment for categorical data called the weighted L measure, which is constructed from the posterior predictive distribution of the data. The measure is based on weighting the observations according to the sampling variance of their future response vector. The weight component in the weighted L measure plays the role of a penalty term in the criterion, in which a greater weight assigned to covariate values implies a greater penalty term on the dimension of the model. A detailed justification is provided for such a weighting procedure and several theoretical properties of the weighted L measure are presented for a wide variety of discrete data models. For these models, we examine properties of the weighted L measure, and show that it can perform better than the unweighted L measure in a variety of settings. In addition, we show that the weighted quadratic loss L measure is more attractive than the unweighted L measure and the deviance loss L measure for categorical data. Moreover, a calibration for the weighted L measure is motivated and proposed, which allows us to compare formally the L measure values of competing models. A detailed simulation study is presented to examine the performance of the weighted L measure, and it is compared to other established model-selection methods. Finally, the method is applied to a real dataset using a bivariate ordinal response model. Copyright Biometrika Trust 2004, Oxford University Press.