Measuring Large-Scale Market Responses from Aggregated Sales - Regression Model for High-Dimensional Sparse Data -

In this article, we propose a regression model for high-dimensional sparse data from storelevel aggregated POS systems. The modeling procedure comprises two sub-models--topic model and hierarchical factor regression model--that are applied sequentially not only for accommodating high dimensionality and sparseness but also for managerial interpretation. First, the topic model is applied unusually to aggregated data to decompose the daily aggregated sales volume of a product into sub-sales for several topics by allocating each unit sale ("word" in text analysis) in a day ("document") into one of topics based on joint purchase information. This stage reduces the dimension of data inside topics because topic distribution is not uniform and product sales are allocated mostly into smaller numbers of topics. Next, the market response regression model in the topic is estimated by using information about other items in the same topic. That is, we construct a topic-wise market response function by using explanatory variables not only of itself but also of other items belonging to the same topic. Additional reduction of dimensionality remains necessary for each topic, and we propose a hierarchical factor regression model based on canonical correlation analysis for original high-dimensional sample spaces. Then we discuss feature selection based on a credible interval of parameters' posterior density. Empirical study shows that (i) our model has the advantage of managerial implications obtained from topic-hierarchical factor regression defined according to their contexts, and (ii) it offers better fit than does conventional category regressions in-sample as well as out of sample.